Malware analysis with VM instrumentation, WMI, winexe, Volatility and Metabrik

In this article, we will show how to take advantage of Metabrik to automate some malware analysis tasks. The goal will be to execute a malware in a virtual machine (VM), just after you saved a snapshot of Windows operating system. In our example, this snapshot only includes running processes, but you will see you can do more than just that. Here, we introduce remote::wmi, remote::winexe and system::virtualbox Briks.

We will also introduce the forensic::volatility Brik which can help you perform dynamic malware analysis and extract IOCs, for instance.

Tip: you can use <tab> keystroke to complete Brik names and Commands while using The Metabrik Shell.

Setting up the environment

wmic and winexe are programs that have to be compiled by yourself. Fortunately, Metabrik makes this process as easy as running the install Command. Since wmic and winexe programs ship with the same software suite, you just have to run install Command for one of remote::wmi or remote::winexe Briks. We don’t run the install Command with system::virtualbox Brik, because we suppose you already have some VitualBox VMs installed.

use brik::tool
use remote::wmi
use remote::winexe
use forensic::volatility
help remote::wmi
help remote::winexe
help forensic::volatility
run brik::tool install_needed_packages remote::wmi
run brik::tool install_needed_packages remote::volatility

screenshot-00002

Your VM also has to be configured to allow WMI accesses for a given user, and have the WINEXESVC service started. Some help on how to do that can be found in remote::wmi and remote::winexe Briks source code.

Starting a VM and taking a snapshot

Our environment is up and running. Let’s start a VM and take a snapshot before we execute a malware within it remotely. For the purpose of this exercise, the malware will simply be calc.exe program.

use system::virtualbox
help system::virtualbox
run system::virtualbox list

screenshot-00003

Let’s start our Windows machine in headless mode: we don’t want to speak with this kind of GUI.

set system::virtualbox type headless
run system::virtualbox start 602782ec-40c0-42ba-ad63-4e56a8bd5657
run system::virtualbox snapshot_live 602782ec-40c0-42ba-ad63-4e56a8bd5657 "before calc.exe"

screenshot-00004

 

I know the IP address of the machine, but you could have found it by using ARP scanning on vboxnet0 interface thanks to the network::arp Brik.

my $win = '192.168.56.101'
my $user = 'Administrator'
my $password = 'YOUR_SECRET'
set remote::wmi host $win
set remote::wmi user $user
set remote::wmi password $password
set remote::winexe host $win
set remote::winexe user $user
set remote::winexe password $password
run remote::wmi get_win32_process
for (@$RUN) {
print $_->{Name}."\n";
}

You should see no calc.exe right now.

screenshot-00005

Now, launch the calc.exe program and search in the process list if you can find it. Note that you will have to run Ctrl+C keystrokes because the program will block here. But calc.exe should still be running on the remote host.

run remote::winexe execute "cmd.exe /c calc.exe"
run remote::wmi get_win32_process
my @processes = map { $_->{Name} } @$RUN
my $found = grep { /calc.exe/ } @processes

In the below screenshot, you will see 2 as a result to the grep command. That’s because we ran two times the execute Command with calc.exe during our testing.

screenshot-00006

Now, we will restore the VM to its default state, when calc.exe “malware” was not yet run.

run system::virtualbox stop 602782ec-40c0-42ba-ad63-4e56a8bd5657
run system::virtualbox snapshot_restore 602782ec-40c0-42ba-ad63-4e56a8bd5657 "before calc.exe"
run system::virtualbox start 602782ec-40c0-42ba-ad63-4e56a8bd5657
run remote::wmi get_win32_process
my @processes = map { $_->{Name} } @$RUN
my $found = grep { /calc.exe/ } @processes

screenshot-00008
All clear. No more calc.exe process.

You spoke about Volatility?

Yes. And that’s where it starts to get interesting. You can do the same processes analysis with Volatility (and of course much more). To use Volatility, you need a dump of the system’s memory. To acquire this dump, it’s as simple as using the system::virtualbox dumpguestcore Command. Then, you have to extract the memory dump that is part of the generated core file. You will use the extract_memdump_from_dumpguestcore Command.

Then, you will be able to perform forensic stuff on this memory dump, for instance to search if calc.exe has been popped. If you go back to the original subject -malware analysis-, you will find the Volatility is the tool of choice to check what a malware you just run with remote::winexe Brik did to processes, network handles or registry. That’s a perfect combination of tools to extract IOCs from a malware.

run system::virtualbox dumpguestcore 602782ec-40c0-42ba-ad63-4e56a8bd5657 dump.core
run system::virtualbox extract_memdump_from_dumpguestcore dump.core dump.volatility

screenshot-00010

We have a dump usable by Volatility. Let’s dig into it with forensic::volatility Brik:

use forensic::volatility
set forensic::volatility input dump.volatility
run forensic::volatility imageinfo
set forensic::volatility profile $RUN->[0]
run forensic::volatility pslist

screenshot-00011

screenshot-00012

And voilà.

A feature of WINEXESVC: get a remote shell on Windows

One last screenshot in regards to remote::winexe Brik: how to get a Windows remote shell:

run remote::winexe execute cmd.exe

screenshot-00009

Conclusion

We have seen that we can easily perform malware analysis on a Windows machine by using a combination of Briks. By combining features of different tools (VirtualBox, winexe and Volatility) we can, for instance, analyse consequences of running a malware on a machine. Extracting IOCs from a malware is something useful if you want to find which machines were infected on your information systems from a particuliar sample. You could then use remote::wmi Brik to scan your network for these specific patterns.

Extracting IOCs is a huge topic in itself, and we just scratched the surface here by using a dynamic method associated with a “scapegoat” VM. Another way of extracting IOCs is to use static analysis, but that’s a complete different story.

We urge you to play with Volatility (and of course Metabrik), you will see how powerful it could be. Enjoy.

Solving a root-me forensic challenge with Metabrik and Scalpel

I recently discovered the wonderful world of forensic and challenges (read: today). So I decided to add some new Briks just to solve some of them. Let’s dig into the “Find the catroot-me challenge step-by-step. I will also show how I improved the Scalpel tool by wrapping it with other Briks.

Getting the files

Well, you have to register to http://www.root-me.org/ to get access to challenge files. The one from this Metabrik Example is called “Find the cat“, or “Trouvez le chat” in French. Once downloaded and extracted, you got these files:

cd /home/gomor/hgwork/metabrik/challenges/trouvez-le-chat/
ls
my $files = $RUN

First things to do is to check for MIME types of these files. You can also check for the MAGIC types:

use file::type
run file::type get_mime_type $files
run file::type get_magic_type $files

screenshot-001

You have a txt file (you should read it for the story behind the cat theft) and a gzip file. Let’s uncompress this one:

use file::compress
run file::compress uncompress $files->[0]
my $file = $RUN
run file::type get_mime_type $file
run file::type get_magic_type $file

screenshot-002

Wonderful. It appears to be some kind of filesystem. Let’s analyse that with Scalpel, a filesystem image forensic tool written in Python.

Introducing Scalpel

To use Scalpel, you usually have to create a scalpel.conf file containing metadata on how to extract (or carve) files. For instance, if you want to find and extract ZIP files, you search for the string PK\x03\x04 in a bytestream.

If you want to only check for some files (say odt files), you have to comment out all the lines from that conf file, except the one for odt documents. If you want to search for all file formats, you have to uncomment all the lines. That is the first thing I changed when writing the forensic::scalpel Brik: use a Command to help you generate the configuration file with only what you want to search for.

The other limitation of the tool is its inhability to identify extracted files using the libmagic (check for MIME types or MAGIC types). Thus, I added this feature within forensic::scalpel Brik to just do that: separate verified files from unverified ones.

So, let’s dig into this new Brik. It is made upon 4 existing Briks: shell::command, file::find, file::text and of course the file::type one. As you can see, new Briks can be written with already existing Briks.

We will use it to extract files from the challenge (remember, you have to find where is the cat). Because I already know the result (spoiler), I will only search for odt files from the filesystem image ch9:

use forensic::scalpel
run forensic::scalpel generate_conf "[ 'odt' ]"
run forensic::scalpel scan $file
my $verified = $RUN->{verified}

We just wanted to keep verified files. Yes, the ones that went through the file::type Brik and its MIME type identification. Read: the feature lacking from Scalpel.

screenshot-004

We now have two odt files, let’s open them with LibreOffice (exercise for the reader).

my $files = join(' ', @$verified)
libreoffice $files

You have a picture of a cat, and a message saying: “Free Alsace, or we kill the cat“. We must locate those miscreants. This is a picture, is there some EXIF metadata? Use the image::exif Brik to discover. But first, we have to extract this picture. odt files are simple ZIP files, we use the file::compress Brik again:

run file::compress uncompress $verified->[0]
my $pic = './Pictures/1000000000000CC000000990038D2A62.jpg'
run file::type get_mime_type $pic

screenshot-005
use image::exif
run image::exif get_metadata $pic

screenshot-006

Lots of metadata. But more interestingly, you have the latitude and longitude of the camera which took the picture. Bingo. Sarge, we found the evil doers, let’s go catch them now. Over.

Conclusion

By using a few Briks, we have shown how to solve a simple challenge. As you can see, you can nearly automate it from the beginning to the end. Exercise yourself with Metabrik by downloading the docker image. Enjoy.

EDIT 2015/12/17: some Commands have moved from system::file to file::type, thus some Commands have been renamed in this post, but screen captures remain the same as before.

Is Telegram using encryption? How to discover it easily by yourself – Part 1

It is said that, by default, messages sent to a contact through Telegram -a messaging application for smartphones- are not encrypted. You have to enter a specific menu named “New Secret Chat” to enable end-to-end encryption. Let’s verify it is indeed the case by using some Briks.
 
Try by yourself using the Docker image:

docker pull metabrik/metabrik
docker run -it metabrik/metabrik

Let’s load some Briks for the work

We will need to perform a Man-in-The-Middle (MiTM) attack on our local network to allow interception of traffic from a smartphone to Telegram servers or remote peers. The network::arp Brik has such a function. We will also need to become a router, or the traffic will be lost: network::route comes to the rescue. Then we will have to analyse the traffic itself, we will use Briks network::read, network::stream and client::whois to locate Telegram IP addresses. We will also use lookup::oui to find a potential smartphone on the network.

use network::arp
use network::route
use network::read
use network::stream
use lookup::oui
use client::whois

screenshot-002
Also, you have to execute an update Command on the lookup::oui Brik so it fetches the file from IEEE organization.

run lookup::oui update

Performing the MiTM attack

We will use ARP poisoning to perform a standard LAN MiTM attack. But we don’t want to poison everyone, we just want to listen to a smartphone traffic. We will use some ARP scanning technics to gather available neighbors, and we will perform a lookup on the MAC address to retrieve the vendor. This information will directly lead us to a smartphone.

run network::arp scan
my $scan = $RUN
my $mac = [ keys %{$RUN->{by_mac}} ]
run lookup::oui from_hex $mac->[0]
run lookup::oui from_hex $mac->[1]

screenshot-004
Looks like we have found a Motorola smartphone. Perfect target for us. To gather its IP address, just issue a Command to ask data from a saved variable:

my $victim = $scan->{by_mac}{"5c:51:88:XX:XX:XX"}

screenshot-005
Now, we want to intercept traffic between the victim and the Internet. Thus, we will attack the gateway. We have to find its IP address, configure our host as a network router, and we will be ready to perform the ARP poisoning:

my $victim = "192.168.1.20"
run network::route default_ipv4_gateway
my $gateway = $RUN
run network::route enable_router_ipv4
run network::arp full_poison $victim $gateway

screenshot-006

Conclusion

We have seen how to scan the local network in search for a specific device and how to launch a Man-in-The-Middle attack. This concludes the first part of this article. You may think it is a little bit short, but you will probably be eager to read the next part :)

Metabrik Core And Repository 1.10 Released

Following our lightning talk from Hack.lu 2015 conference, we are proud to release the version 1.10 of Metabrik Core and Repository. Update using Mercurial or follow the installation procedure.

You can find the few slides which were presented at the following link.

Lots of new awesome Briks

We added many Briks for this new release, here is the description for them:

  • api::bluecoat: play with Bluecoat REST API
  • api::splunk: play with the Splunk REST API
  • api::virustotal: play with Virustotal REST API
  • client::udp: a UDP socket client (UDP netcat)
  • client::ssl: check various stuff about a SSL/TLS connection
  • client::rest: the base REST client for use with Briks from API Category
  • client::rsync: a wrapper around rsync program
  • client::twitter: a Twitter client
  • database::mysql: interract with MySQL databases
  • file::dump: read and write dump files
  • file::hash: genrated various digests from files
  • file::ole: play with Microsoft files that embed OLE components
  • lookup::iplocation: geolocation for IP addresses
  • string::ascii: convert ASCII characters
  • string::csv: encode/decode CSV strings
  • string::hostname: parse a FQDN
  • string::regex: experiment with regexes
  • system::freebsd::pf: control Packet Filter
  • system::freebsd::jail: control jails

Just type help <Brik> to know more:

Meta:~> use string::regex 
[*] core::shell: use: Brik [string::regex] success
Meta:~> help string::regex 
[+] run string::regex encode <$regex|$regex_list>

Complete list of changes

Core

1.10 Tue Oct 27 20:13:36 CET 2015
   - FEATURE: core::context: allows to pass complex structs arguments to run and set Commands
     Example: run network::arp scan $info->{subnet}
   - FEATURE: core::context: allows also to execute Perl code within an Argument of a
     run Command
     Example: run client::dns ptr_lookup "[ map { @$_ } values %$RUN ]"
   - FEATURE: core::shell: allows to complete aliases (can be disabled via
     aliases_completion Attribute
   - FEATURE: shell::command: use_sudo Attribute to launch sudo on executing external command
   - FEATURE: shell::command: file globbing enabled with capture Command
   - UPDATE: moved attributes_default() from brik_use_properties to brik_properties when
     there is no need to use $self. It allows instanciated Attributes inheritage to work.
   - UPDATE: shell::command: do not print STDERR when using capture Command when there is no
     STDERR string captured.
   - new: shell::command: execute Command to use capture_mode Attribute to launch either
     capture or system Command
   - bugfix: core::context: save_state to use Metabrik brik_attributes Command to correctly
     retrieve all Brik Attributes even those inherited
   - bugfix: core::shell: display error on chdir() failure
   - bugfix: core::shell: escapes " character when executing a multiline Perl/Metabrik Code
             example:
             my $test = 'root'
             for (0..1) {
                'run shell::command system "ls /$test"'
             }
   - bugfix: Metabrik: error checking within new_from_brik_init Command
   - bugfix: Metabrik: logging correctly on class calls to _log_*()

Repository

- bugfixes and new Briks

20151011
   AFFECT: network::arp

   - network::arp scan Command now returns a hashref with results sorted
     with keys named by_mac, by_ipv4 and by_ipv6

20151003
   AFFECT: network::rsync

   - network::rsync renamed to client::rsync

20150418
   AFFECT: crypto::x509

   - Argument order changed for ca_sign_csr and cert_verify Commands

20150322
   AFFECT: file::csv

   - removed get_col_by_name and get_col_by_number obsolete Commands

IP location with Metabrik: different ways of doing it

We just released a new Brik: lookup::iplocation. Thanks to this Brik, you will be able to easily geo-locate any IP address. Well, it has to be in MaxMind’s database for that to work, of course. In this post, we propose to explore different ways of using a Metabrik Brik with this lookup::iplocation example.

Using The Metabrik Shell

Of course, the simplest way of doing is using The Metabrik Shell (which is just a simple Brik called core::shell). A screenshot is what best demonstrates the power of Metabrik.

The first call to ‘use lookup::location‘ fails because we currently lack one Perl module: Geo::IP. Metabrik tells you so and you can install it by using the perl::module Brik.

Then, you want to update the Geo::IP database. If you don’t, you will not have the required data to perform the lookup. You have to call ‘run lookup::location update‘.

Finally, you can geo-locate the IP address by calling ‘run lookup::location from_ip <IP>‘.

iplocation01

Using a Metabrik script

Before being able to execute Metabrik scripts (.meta files), you have to initiate the .metabrik_script_rc file. Just edit it using vi, for instance, and put in the lines below:

vi ~/.metabrik_script_rc
set core::shell echo 0
my $home = $ENV{HOME}
my $repository = "$HOME/metabrik/repository/lib"
push @INC, $repository
run core::context update_available

Now you are ready to write your very own Metabrik script. Create one using vi and put in this content:

vi iplocation-script.meta
use lookup::iplocation
if ($ERR) { exit }

run lookup::iplocation update
if ($ERR) { exit }

run lookup::iplocation from_ip 106.219.10.202
if ($ERR) { exit }

use Data::Dumper
print Dumper($RUN)

You can mix Metabrik Commands with Perl code. Wonderful, isn’t it? Ah, maybe you want to execute it:

metabrik --script iplocation-script.meta

The third way: use a Perl script

And finally, this is Perl. You can use a Brik as a plain Perl module if you want to. Try that:

vi iplocation-standalone-brik.pl
#!/usr/bin/perl
use strict;
use warnings;

use Data::Dumper;
use Metabrik::Core::Context;
use Metabrik::Lookup::Iplocation;

my $context = Metabrik::Core::Context->new;
$context->brik_init or die("[FATAL] context init failed\n");

my $li = Metabrik::Lookup::Iplocation->new_from_brik_init($context)
   or exit(1);

$li->update
   or exit(2);

my $info = $li->from_ip("104.47.125.219")
   or exit(3);

print Dumper($info);

Conclusion

You see you can play with Metabrik in different ways. Even though Metabrik has been thought as a shell from the beginning, you are not bound to it in any way. And you, what is your favorite way of using Briks?

Metabrik Core And Repository 1.08 Released

A new version of Metabrik Core and Repository is available. Update using Mercurial or follow the installation procedure.

Changes

Core

1.08 Thu Mar 19 06:48:34 CET 2015
 - FEATURE: core::shell: run executable commands found in PATH through system Command
 - UPDATE: shell::command: now use IPC::Run3 to capture shell commands output
 - update: shell::command: system Command now returns $? on success
 - update: Metabrik: display every missing items from brik_require_*_check() before erroring
 - update: new dependance on IPC::Run3

1.07 Sun Mar 8 17:52:37 CET 2015
 - bugfix: shell::history: correctly writes $* variables when calling history Commands
 - new: shell::rc: create an alias to make it easy to switch to root from Shell
 - new: brik::search and perl::module Briks integrated in main distribution
 - remove: no more metabrik-cpanm, enforce use of standard cpanm

Repository

- bugfixes and new Briks

20150316
 AFFECT: http::proxy, iana::countrycode, network::arpdiscover

 - http::proxy Brik renamed to proxy::http
 - iana::countrycode Brik renamed lookup::countrycode
 - network::arpdiscover removed: merged with network::arp

20150311
 AFFECT: client::www

 - post Command returns an HASHREF i/o of WWW::Mechanize object

20150309
 AFFECT: network::portscan

 - synscan Command renamed to tcp_syn

 

Exploiting ElasticSearch RCE For CVE-2015-1427

I told you so: it is a work for Metabrik. While the main target for Metabrik is not to write exploits (you have Metasploit for that), you can still write Briks within the Audit Category. Based on the exploit provided XiphosResearch, we wrote a Command to verify if an ElasticSearch target is vulnerable, and another Command to exploit the issue to execute commands on the target.

Loading the Brik

As always, if a Brik is not loaded yet, you have to do it. Then, the first thing to do is to ask for help, or how to use the Brik, which Attributes can be set and which Commands can be run.

screenshot

Using a check or an exploit Command

To test a target, you can either use the check_cve_2015_1427_rce  Command or use the exploit_cve_2015_1427_rce Command with an innocuous command. We recommand to use the check one, which is actually made to be innocuous.

To use the check or exploit one, you just have to use the run Metabrik Command with the name of a Command, and add Arguments to it. Some Arguments can be set globally for the Brik: here you may use the set Metabrik Command to set uri Attribute.

Note: And don’t forget to use the <Tab> key to perform completion on every Command, and use the <Up> key to recall previous ones.

Example:

screenshot

Our target is vulnerable. Too bad, but it is an exploit lab, it exists for that.

Exploiting the issue to execute commands

But well, if you are here, it is probably because you want to exploit a Remote Command Execution within ElasticSearch. With Metabrik, it is as easy as:

screenshot

We even added some “post intrusion” work, like downloading a file.

A key feature of Metabrik

Yes, one of the key feature of Metabrik is to assemble Briks together to execute a complete scenario. We have shown how to use Briks, especially the one on exploiting a vulnerability within a product and, after this exploitation process, we have shown that we can use a special $RUN variable with another Brik.

Every:

  • run Command sets the global variable $RUN
  • set Command sets the global variable $SET
  • get Command sets the global variable $GET

Thanks to the $RUN variable, you can chain the use of Briks. In fact, Briks within Metabrik may also be written with other Briks. For instance, audit::elasticsearch Brik relies on:

Here, we have chained a few Briks, and one of them allowed to save a remote file to a local file (file::text). You have more global variables, but that’s enough for today. Follow us on twitter @Metabrik.

 

 

Why Writing A TCP SYN Scanner In Perl Can Be Efficient

Everyone wants to write (or already did) its own TCP SYN scanner. Why? Because it is a fun exercise which will teach you a lot of things, like what raw sockets are and, more importantly, how to build packets.

Perl for the task

Some may argue that using Perl would be inefficient for such a task. Plain false. You can use Perl/XS to write part(s) of a program that require performance. That’s what we did with Net::Write::Fast: a module able to send packets over the network at speed of wires.

Design goals behind Net::Write::Fast

A few design goals were followed when writing this module:

  • TCP SYN scanner working over IPv4 and IPv6
  • Portable on at least Linux, FreeBSD and MacOS X
  • Perform a scan in a predictable way: we know when it will finish
  • Send packets over a transport layer socket

The transport layer socket

Usually, people tend to write packets using raw sockets directly from the network layer, requiring them to also craft the IP layer. And that’s a total waste of code and time. You can send your TCP or UDP packets without having to craft the IP header; no need to use the include IP header socket option.

To make it short, opening a raw socket within Net::Write::Fast was as simple as:

int fd;
fd = socket(v6 ? AF_INET6 : AF_INET, SOCK_RAW, IPPROTO_TCP);

The TCP header

So you did open a raw socket at the transport layer, now you just need to write the TCP layer. You don’t have to care about IP layer at all, and the differences of endianess from OS to OS for some of its fields (header length, for instance). Well, you still have to write the pseudo header so you can compute TCP checksum, but that’s another story.

Glueing C code to Perl

Full Perl/XS code can be found here. It is exactly like standard C code. But for the C function to be callable from Perl, we need to add a binding. It is a little bit tricky when you want to do it right. The prototype of the function is:

int
l4_send_tcp_syn_multi(src, dst, ports, pps, n, v6, warnings = NO_INIT)
      char *src
      SV   *dst
      SV   *ports
      int   pps
      int   n
      int   v6
      int   warnings

A forked sender process and a listener

Now everything is in place to send packets at the speed of wires. We still have to think about listening for answers and parsing them. Here, no need for Perl/XS for the listening task: you have Net::Frame::Dump module which is a pure Perl one.

Now, we need proof

Here is a screenshot from a TCP SYN scan of the TOP 1000 most common ports on the local network (256 hosts). We configure the scanner to send 2 times a probe, at a rate of 100 000 packets per second. If you do your math, you will see it should take 5 seconds and consume around 8 MB of bandwidth per second. One packet takes 20 bytes for IP header followed by 60 bytes for the TCP header plus its full options.

Of course, you can send at a greater speed, or change the number of tries to tweak your scanning process. Screenshot:

screenshot

So, it took a little bit more than the 5 seconds estimation, but you have 2 or 3 seconds needed by the program to load and compile, and to wait at the end of sending process to gather remaining replies.

But what about IP source address spoofing?

Some may argue that we cannot spoof source IP address using transport layer sending. And that’s just wrong. You can do it by just adding an alias to your network interface, and using that as the source address for sending packets. It will be used right away by your transport layer raw socket.

Tweaking your TCP/IP stack

Under GNU/Linux, don’t forget to increase your send buffers. Or you will have the following error:

WARNING: Net::Write::Fast: sendto: ENOBUFS, sleeping for 1 second

# Increase send buffer to 100 MB
sysctl -w net.core.wmem_max=109051904
sysctl -w net.core.wmem_default=109051904

Conclusion

Writing a TCP SYN scanner in Perl is reliable, fast, and predictable. And of course, there is now a Brik for that within Metabrik, it is named network::portscan. Happy scanning, and don’t forget to follow us on Twitter: @metabrik.

Metabrik Core And Repository 1.06 Released

A new version of Metabrik Core and Repository is available. Update using Mercurial or follow the installation procedure.

Changes

Core

1.06 Fri Feb 27 07:17:59 CET 2015
 - bugfix: shell::command: go trough PATH to find a cmd to run (like less PAGER)
 - bugfix: core::shell: on SIGINT handling, now allows to break multiline and run Commands
 - bugfix: core::shell: allow user to get out of multiline mode by hitting Ctrl+C
 - bugfix: core::shell: better management of Metabrik Commands in multiline mode
 => you can put Metabrik Commands within single quotes anywhere
 - update: shell::script: load Command can take an optional file parameter
 - update: metabrik, shell::rc/script, core::shell: easier handling of rc file loading
 - new: metabrik: --script-rc argument to load a specific rc file for scripts

Repository

- bugfixes and new Briks
- see UPDATING file for changes since your last update

 

Playing With Ten Million Passwords And Logins

A researcher has released a database of ten million logins and passwords. We will show how we can play with it with Metabrik.

Fetching the data

You have to find the data by yourself, go check it from Mark Burnett post: Today I Am Releasing Ten Million Passwords.

Loading Metabrik Briks

Note: You will need Metabrik 1.06+ to run this example. Check it out via the Mercurial repository.

Note: When you are using the Shell, do not forget that you can use key to auto-complete Commands and Variables.

The file is a zip archive, so you will use the file::compress Brik. The uncompressed file being a text file, you will need to parse it in a low-level way: file::read Brik will be used.

Meta:~> use file::compress
[*] core::shell: use: Brik [file::compress] success
Meta:~> use file::read
[*] core::shell: use: Brik [file::read] success

 

Let’s read the data and do some statistics

Uncompress the archive

So, we have first to uncompress the archive. By default, uncompressed data will be put in the Brik home directory: ~/metabrik/file-compress.

Meta:~> run file::compress unzip ~/Downloads/10-million-combos.zip
"/home/gomor/metabrik/file-compress"
Meta:~> l /home/gomor/metabrik/file-compress/10-million-combos.txt
[
  "-rw-r--r-- 1 gomor gomor 194130539 Feb  9 12:59 /home/gomor/metabrik/file-compress/10-million-combos.txt",
]

 

Read and parse

# No need to print anything by default during processing
set core::shell echo 0

# Configure the file::read Brik
set file::read input ~/metabrik/file-compress/10-million-combos.txt
set file::read encoding ascii
set file::read strip_crlf 1

# Start using it
run file::read open

Now, file is open, and we can start reading and parsing it. What we want here is to perform some statistics, like getting the top used passwords. To that end, we will mix Metabrik Commands with Perl code. The Shell allows you to do that in a simplified way that going to write a Perl script or program.

my $stats = {}   # Declare a variable to store results

# Get access to Perl object, this is a performance hack for power users.
run file::read brik_self
my $read = $RUN

my $count = 0
# Alternatively (but slower), we could have replaced the following line by:
# while ('run file::read read_line') {
while (my $line = $read->read_line) {
   my ($l, $p) = split(/\t/, $line);
   $stats->{$p}++;
   $count++;
   if (! ($count % 10_000)) {  # Print count every 10_000 lines
      print "$count\n";
   }
   last if $read->eof;
}

# Get top 20 passwords
my $top = 20;
for my $k (reverse sort { $stats->{$a} <=> $stats->{$b} } keys %$stats) {
   print "$k => ".$stats->{$k}."\n";
   $top--;
   last if $top 0;
}

 

And top 20 is…

  • 123456 => 55893
  • password => 19580
  • 12345678 => 13582
  • qwerty => 13137
  • 123456789 => 11696
  • 12345 => 10938
  • 1234 => 6432
  • 111111 => 5682
  • 1234567 => 4796
  • dragon => 3927
  • 123123 => 3845
  • baseball => 3565
  • abc123 => 3511
  • football => 3494
  • monkey => 3246
  • letmein => 3118
  • 696969 => 3050
  • shadow => 2956
  • master => 2931
  • 666666 => 2905

And what about a Script?

Yes, you can write a Metabrik Script to perform this task. Just copy and past this data into a file named top20-password.brik for instance:

use file::read
set file::read input ./10-million-combos.txt
set file::read encoding ascii
set file::read strip_crlf 1
run file::read open

set core::log level 0

run file::read brik_self
my $read = $RUN

my $stats = {}
my $count = 0
while (my $line = $read->read_line) {
   my ($l, $p) = split(/\t/, $line);
   $stats->{$p}++;
   $count++;
   if (! ($count % 10_000)) {
      print "$count\n";
   }
   last if $read->eof;
}

my $top = 20;
for my $k (reverse sort { $stats->{$a} <=> $stats->{$b} } keys %$stats) {
   print "$k => ".$stats->{$k}."\n";
   $top--;
   last if $top  0;
}

 

And run the script:

$ metabrik --script top20-password.brik

 

That’s all for today. Follow @metabrik on twitter.