A researcher has released a database of ten million logins and passwords. We will show how we can play with it with Metabrik.

Fetching the data

You have to find the data by yourself, go check it from Mark Burnett post: Today I Am Releasing Ten Million Passwords.

Loading Metabrik Briks

Note: You will need Metabrik 1.06+ to run this example. Check it out via the Mercurial repository.

Note: When you are using the Shell, do not forget that you can use key to auto-complete Commands and Variables.

The file is a zip archive, so you will use the file::compress Brik. The uncompressed file being a text file, you will need to parse it in a low-level way: file::read Brik will be used.

Meta:~> use file::compress
[*] core::shell: use: Brik [file::compress] success
Meta:~> use file::read
[*] core::shell: use: Brik [file::read] success

Let’s read the data and do some statistics

Uncompress the archive

So, we have first to uncompress the archive. By default, uncompressed data will be put in the Brik home directory: ~/metabrik/file-compress.

Meta:~> run file::compress unzip ~/Downloads/10-million-combos.zip
"/home/gomor/metabrik/file-compress"
Meta:~> l /home/gomor/metabrik/file-compress/10-million-combos.txt
[
  "-rw-r--r-- 1 gomor gomor 194130539 Feb  9 12:59 /home/gomor/metabrik/file-compress/10-million-combos.txt",
]

Read and parse

# No need to print anything by default during processing
set core::shell echo 0

# Configure the file::read Brik
set file::read input ~/metabrik/file-compress/10-million-combos.txt
set file::read encoding ascii
set file::read strip_crlf 1

# Start using it
run file::read open

Now, file is open, and we can start reading and parsing it. What we want here is to perform some statistics, like getting the top used passwords. To that end, we will mix Metabrik Commands with Perl code. The Shell allows you to do that in a simplified way that going to write a Perl script or program.

my $stats = {}   # Declare a variable to store results

# Get access to Perl object, this is a performance hack for power users.
run file::read brik_self
my $read = $RUN

my $count = 0
# Alternatively (but slower), we could have replaced the following line by:
# while ('run file::read read_line') {
while (my $line = $read->read_line) {
   my ($l, $p) = split(/\t/, $line);
   $stats->{$p}++;
   $count++;
   if (! ($count % 10_000)) {  # Print count every 10_000 lines
      print "$count\n";
   }
   last if $read->eof;
}

# Get top 20 passwords
my $top = 20;
for my $k (reverse sort { $stats->{$a} <=> $stats->{$b} } keys %$stats) {
   print "$k => ".$stats->{$k}."\n";
   $top--;
   last if $top == 0;
}

And top 20 is…

• 123456 => 55893
• password => 19580
• 12345678 => 13582
• qwerty => 13137
• 123456789 => 11696
• 12345 => 10938
• 1234 => 6432
• 111111 => 5682
• 1234567 => 4796
• dragon => 3927
• 123123 => 3845
• baseball => 3565
• abc123 => 3511
• football => 3494
• monkey => 3246
• letmein => 3118
• 696969 => 3050
• shadow => 2956
• master => 2931
• 666666 => 2905

And what about a Script?

Yes, you can write a Metabrik Script to perform this task. Just copy and past this data into a file named top20-password.brik for instance:

use file::read
set file::read input ./10-million-combos.txt
set file::read encoding ascii
set file::read strip_crlf 1
run file::read open

set core::log level 0

run file::read brik_self
my $read = $RUN

my $stats = {}
my $count = 0
while (my $line = $read->read_line) {
   my ($l, $p) = split(/\t/, $line);
   $stats->{$p}++;
   $count++;
   if (! ($count % 10_000)) {
      print "$count\n";
   }
   last if $read->eof;
}

my $top = 20;
for my $k (reverse sort { $stats->{$a} <=> $stats->{$b} } keys %$stats) {
   print "$k => ".$stats->{$k}."\n";
   $top--;
   last if $top == 0;
}

And run the script:

$ metabrik --script top20-password.brik

That’s all for today. Follow @metabrik on twitter.