This article presents several new open source frameworks meant to simplify static file scanning for malware analysis and incident response: MASTIFF, Viper, IRMA and a few others. Their goal is to provide an extensible framework to integrate many existing scanning tools.
Note: This article is about tools running locally on your own system for static file analysis. I will not describe online file analysis services such as VirusTotal or Wepawet, which have already been well covered (for example here and here). It is not either about dynamic malware analysis tools such as Cuckoo Sandbox (see here).
Please do not hesitate to contact me if you have comments or if you know another tool similar to the ones described in this article.
There is a growing number of tools developed for malware analysis. For static analysis, we have now a large choice of tools to hash files, to identify their type, to parse their content, to extract metadata and main characteristics, to identify anomalies or signs of malicious content, to detect and extract embedded files, etc. To analyze a single file, it may become a bit tedious to run all the relevant tools and store the results in a convenient location.
The idea of a file scanning framework is to integrate all these tools into a single interface ready to use to simplify the work of the analyst, to automate parts of the work, and to store the results in a structured way (e.g. a database) for further analysis across a collection of samples. Ideally, the framework should be able to detect file types in order to run the corresponding tools in a smart way (e.g. PDF tools for PDF, PE tools for PE executables, etc). It should also handle archives, containers and embedded files automatically. And last but not least, such a framework should have a simple plugin system, so that the community could easily contribute new scanning features.
From the home page: "MASTIFF is a static analysis framework that automates the process of extracting key characteristics from a number of different file formats. To ensure the framework remains flexible and extensible, a community-driven set of plug-ins is used to perform file analysis and data extraction. While originally designed to support malware, intrusion, and forensic analysis, the framework is well-suited to support a broader range of analytic needs. In a nutshell, MASTIFF allows analysts to focus on analysis rather than figuring out how to parse files."
Main features and characteristics:
Plugins available in the current version 0.6.0 as of 11 July 2014:
MASTIFF screenshot from http://digital-forensics.sans.org/blog/2013/05/07/mastiff-for-auto-stati...
Links:
Other articles and related tools:
Viper is a "framework to store, classify and investigate binary files of any sort". It provides a command-line shell with a number of commands to open and store files, to analyze them using various tools and plugins written in Python, and to perform other actions. All results and files are stored in a database to keep them organized for further analysis such as finding similar samples. Unlike MASTIFF, the analysis is not automated, each action is launched by typing a simple command such as "yara" or "pe resources". The tools are well integrated in the shell and the real benefit is the database.
Main features and characteristics:
Plugins available in the current version 1.0 as of 20 July 2014:
Viper screenshot from http://viper.li/
Links:
Other articles and related tools:
From the home page: "IRMA intends to be an open-source platform designed to help identifying and analyzing malicious files. An important value with IRMA comes from you keep control over where goes / who gets your data. Once you install IRMA on your network, your data stays on your network. Each submitted files is analyzed in various ways. For now, we focus our efforts on multiple anti-virus engines, but we are working on other "probes" (feel free to submit your own)."
In short IRMA looks like an open-source VirusTotal-like system running on local servers, with the possibility to extend it like MASTIFF and Viper.
Main features and characteristics:
Plugins available in the current version as of 21 July 2014:
IRMA screenshot from http://irma.quarkslab.com/preview.html
Links:
Workbench is an open-source framework to store and analyze all sorts of data related to an incident (e.g. files and PCAPs). The user interface is based on IPython notebooks. Workbench seems to have a lot of very interesting features similar to the other frameworks above. But to be fair, the current documentation is still a work in progress and I have not spent enough time trying it or digging into the code yet.
Note from the authors: "The project is new and looking for contributors and alpha users!".
Links:
This section lists other open-source frameworks providing similar scanning features, but with different purposes. Part of their code could be reused for malware analysis frameworks.
Ragpicker is a python tool to crawl websites providing malware samples, download them to build a collection, run a number of analysis plugins, and generate reports. Analysis plugins are quite similar to MASTIFF and Viper.
Plugins in current version 0.05.2 as of 14/07/2014:
ExeFilter is a file scanning and cleaning framework developed in python, extensible with plugins. Its main purpose is to filter incoming files from removable devices, e-mails or web browsing, according to a configurable white list of allowed formats. It also cleans file formats that may embed active content (HTML, PDF, MS Office, etc), and processes archives recursively.
Plugins in current version 1.1.4-alpha6 as of 14/07/2014:
Please do not hesitate to contact me if you have comments or if you know another tool similar to the ones described in this article.