Metabrik Core And Repository 1.06 Released

A new version of Metabrik Core and Repository is available. Update using Mercurial or follow the installation procedure.

Changes

Core

1.06 Fri Feb 27 07:17:59 CET 2015
 - bugfix: shell::command: go trough PATH to find a cmd to run (like less PAGER)
 - bugfix: core::shell: on SIGINT handling, now allows to break multiline and run Commands
 - bugfix: core::shell: allow user to get out of multiline mode by hitting Ctrl+C
 - bugfix: core::shell: better management of Metabrik Commands in multiline mode
 => you can put Metabrik Commands within single quotes anywhere
 - update: shell::script: load Command can take an optional file parameter
 - update: metabrik, shell::rc/script, core::shell: easier handling of rc file loading
 - new: metabrik: --script-rc argument to load a specific rc file for scripts

Repository

- bugfixes and new Briks
- see UPDATING file for changes since your last update

 

Playing With Ten Million Passwords And Logins

A researcher has released a database of ten million logins and passwords. We will show how we can play with it with Metabrik.

Fetching the data

You have to find the data by yourself, go check it from Mark Burnett post: Today I Am Releasing Ten Million Passwords.

Loading Metabrik Briks

Note: You will need Metabrik 1.06+ to run this example. Check it out via the Mercurial repository.

Note: When you are using the Shell, do not forget that you can use key to auto-complete Commands and Variables.

The file is a zip archive, so you will use the file::compress Brik. The uncompressed file being a text file, you will need to parse it in a low-level way: file::read Brik will be used.

Meta:~> use file::compress
[*] core::shell: use: Brik [file::compress] success
Meta:~> use file::read
[*] core::shell: use: Brik [file::read] success

 

Let’s read the data and do some statistics

Uncompress the archive

So, we have first to uncompress the archive. By default, uncompressed data will be put in the Brik home directory: ~/metabrik/file-compress.

Meta:~> run file::compress unzip ~/Downloads/10-million-combos.zip
"/home/gomor/metabrik/file-compress"
Meta:~> l /home/gomor/metabrik/file-compress/10-million-combos.txt
[
  "-rw-r--r-- 1 gomor gomor 194130539 Feb  9 12:59 /home/gomor/metabrik/file-compress/10-million-combos.txt",
]

 

Read and parse

# No need to print anything by default during processing
set core::shell echo 0

# Configure the file::read Brik
set file::read input ~/metabrik/file-compress/10-million-combos.txt
set file::read encoding ascii
set file::read strip_crlf 1

# Start using it
run file::read open

Now, file is open, and we can start reading and parsing it. What we want here is to perform some statistics, like getting the top used passwords. To that end, we will mix Metabrik Commands with Perl code. The Shell allows you to do that in a simplified way that going to write a Perl script or program.

my $stats = {}   # Declare a variable to store results

# Get access to Perl object, this is a performance hack for power users.
run file::read brik_self
my $read = $RUN

my $count = 0
# Alternatively (but slower), we could have replaced the following line by:
# while ('run file::read read_line') {
while (my $line = $read->read_line) {
   my ($l, $p) = split(/\t/, $line);
   $stats->{$p}++;
   $count++;
   if (! ($count % 10_000)) {  # Print count every 10_000 lines
      print "$count\n";
   }
   last if $read->eof;
}

# Get top 20 passwords
my $top = 20;
for my $k (reverse sort { $stats->{$a} <=> $stats->{$b} } keys %$stats) {
   print "$k => ".$stats->{$k}."\n";
   $top--;
   last if $top == 0;
}

 

And top 20 is…

  • 123456 => 55893
  • password => 19580
  • 12345678 => 13582
  • qwerty => 13137
  • 123456789 => 11696
  • 12345 => 10938
  • 1234 => 6432
  • 111111 => 5682
  • 1234567 => 4796
  • dragon => 3927
  • 123123 => 3845
  • baseball => 3565
  • abc123 => 3511
  • football => 3494
  • monkey => 3246
  • letmein => 3118
  • 696969 => 3050
  • shadow => 2956
  • master => 2931
  • 666666 => 2905

And what about a Script?

Yes, you can write a Metabrik Script to perform this task. Just copy and past this data into a file named top20-password.brik for instance:

use file::read
set file::read input ./10-million-combos.txt
set file::read encoding ascii
set file::read strip_crlf 1
run file::read open

set core::log level 0

run file::read brik_self
my $read = $RUN

my $stats = {}
my $count = 0
while (my $line = $read->read_line) {
   my ($l, $p) = split(/\t/, $line);
   $stats->{$p}++;
   $count++;
   if (! ($count % 10_000)) {
      print "$count\n";
   }
   last if $read->eof;
}

my $top = 20;
for my $k (reverse sort { $stats->{$a} <=> $stats->{$b} } keys %$stats) {
   print "$k => ".$stats->{$k}."\n";
   $top--;
   last if $top == 0;
}

 

And run the script:

$ metabrik --script top20-password.brik

 

That’s all for today. Follow @metabrik on twitter.