Creating and Verifying MD5 Checksums

Creating MD5 Checksums

The md5sum command found on most Unix/Linux operating systems can be used to create MD5 checksums for files or to verify the integrity of files if the MD5 checksums for those files are already available.

Let us suppose you want to create an MD5 checksum file containing the checksums for the binary files my_disk_image-1.iso, my_disk_image-2.iso and my_disk_image-3.iso. You can accomplish this by passing the three files as arguments to the md5sum command and redirecting the output to a text file:

$ md5sum -b my_disk_image-1.iso my_disk_image-2.iso my_disk_image-3.iso > MD5SUM

The "-b" option instructs the md5sum command to treat each file as a binary file. If you are working text with files, you can use the "-t" option. The generated checksum file "MD5SUM" will look similar to the following:

302d1a8fa7e13871d9909947eb23935d *my_disk_image-1.iso
2f5be4a2fe3d80b134aba6c6023eca57 *my_disk_image-2.iso
52237af3336321e0b03586055b8e5d78 *my_disk_image-3.iso

The first 32 characters of each line is the MD5 checksum for the file mentioned on that line. The asterisk that precedes the file name indicates that the file is a binary file.

Directories and Subdirectories

The following command can be used to compute checksums for file in a given directory and it's sub-directories.

find /path/to/the/directory -type f -print0 | xargs -0 md5sum > MD5SUM

This command creates a file called "md5sums" containing the MD5 checksum for all the files in /usr/share/man and its sub-directories.

find /usr/share/man -type f -print0 | xargs -0 md5sum -b > md5sums

As you can see the by the path names, the command has created MD5 hashes for files residing at various levels of the directory tree.

...
3710f7bc99303ceb90a1ae1e75361913 */usr/share/man/fr/man7/backend.7.gz
f3f6fb8a04b9e78971d875ed8645f848 */usr/share/man/fr/man7/filter.7.gz
9caaf4f56d9f2a72ce9fe977c703475c */usr/share/man/man5/sane-dc210.5.gz
f170bb97e4fc6b919426cfecc2ef583b */usr/share/man/man5/faillog.5.gz
...

Verifying a File's Integrity Using it's MD5 Checksum

To generate the MD5 checksum for a file you can do the following:

$ md5sum -b my_disk_image-1.iso
e36e064cf65e4dc62ea279dc860c8f9a *my_disk_image-1.iso

Checking each of the 32 characters of the checksum against the original is tedious. If you already have the original MD5 checksum file, you can perform the following:

$ cat MDSUM
302d1a8fa7e13871d9909947eb23935d *my_disk_image-1.iso
2f5be4a2fe3d80b134aba6c6023eca57 *my_disk_image-2.iso
52237af3336321e0b03586055b8e5d78 *my_disk_image-3.iso

$ ls my*.iso
my_disk_image-1.iso  my_disk_image-2.iso my_disk_image-3.iso

$ md5sum -c MD5SUM 
my_disk_image-1.iso: OK
my_disk_image-2.iso: OK
my_disk_image-3.iso: OK

Here are some status messages you may see:

OK                   - MD5 checksums matched.
FAILED               - Generally means the MD5 checksums did not match.
FAILED open or read  - The file could not be read or is missing.

When Checksums Fail

Here is a case where validating the checksums failed:

my_disk_image-1.iso: OK
my_disk_image-2.iso: FAILED
md5sum: my_disk_image-3.iso: No such file or directory
my_disk_image-3.iso: FAILED open or read
md5sum: WARNING: 1 of 3 listed files could not be read
md5sum: WARNING: 1 of 2 computed checksums did NOT match

In the above scenario:

• my_disk_image-1.iso was identical to the original.
• my_disk_image-2.iso was different from the original.
• my_disk_image-3.iso was missing from the directory.

Beyond MD5

The sha1sum (that is a "one", not lowercase "L") command can be used to create a SHA-1 Checksum. The sha225sum, sha256su, sha384sum, and sha512sum commands compute the 224, 256, 384, and 512 bit (respectively) SHA-2 hashes. The usage and options of these commands are the same as for the m5sum command.

Links

MD5

http://en.wikipedia.org/wiki/MD5

checksum

http://en.wikipedia.org/wiki/Checksum

md5sum

http://en.wikipedia.org/wiki/Md5sum

Block IP Addresses Using iptables

This post describes the steps to block the IP address that is the origin of a brute-force attack against a WordPress blog.

WARNING!

Make sure you DO NOT block your own IP addresses, or apply and global rules which would block the SSH port (port 22).

Identify IP Addresses to Block (Blog Server)

Tail the blog server apache logs:

sudo tail -f /var/log/httpd/access_log

199.188.70.163 - - [12/Oct/2014:16:29:03 +0000] "POST /wp-login.php HTTP/1.0" 200 4432 "-" "-"
37.59.125.22 - - [12/Oct/2014:16:29:03 +0000] "POST /wp-login.php HTTP/1.0" 200 4432 "-" "-"
199.188.70.163 - - [12/Oct/2014:16:29:04 +0000] "POST /wp-login.php HTTP/1.0" 200 4432 "-" "-"
37.59.125.22 - - [12/Oct/2014:16:29:05 +0000] "POST /wp-login.php HTTP/1.0" 200 4432 "-" "-"
37.59.125.22 - - [12/Oct/2014:16:29:05 +0000] "POST /wp-login.php HTTP/1.0" 200 4432 "-" "-"

This shows two ip addresses, 199.188.70.163 and 37.59.125.22 attempting to brute-force the blog's Word Press login page:

It is worth checking the Apache error log:

sudo tail -f  /var/log/httpd/access_log

Blocking IP Addresses Using iptables

List existing iptables rules:

# List iptables rules
sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

The output of the -S switch can be used to re-create the given rule from the command line or a rule script. To block a single IP addresses 199.188.70.163 and 37.59.125.22 do the following:

sudo iptables -A INPUT -s 199.188.70.163 -j DROP
sudo iptables -A INPUT -s 37.59.125.22 -j DROP

Listing the iptables rules once more should show the DROP rules that were added in the previous step:

# List iptables rules
sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 199.188.70.163 -j DROP
-A INPUT -s 37.59.125.22 -j DROP

The two rules that were added have not taken effect yet. To save them do the following:

# Save the iptables rules
sudo /etc/init.d/iptables save

NOTE: The /sbin/service command is available on the blog server, so you can use that to save iptables rules:

sudo /sbin/service iptables save
The service command runs /etc/init.d/ scripts without passing on environment variables defined by the user.

When successfully saved, it should display a message such as:

iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]

Removing iptables Rules

List iptables Rules:

sudo iptables -S

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 199.188.70.0/24 -j DROP
-A INPUT -s 37.59.125.0/24 -j DROP

The following lists the form in a more readable form:

# List rules with line number in verbose mode
sudo iptables -L -n -v --line-numbers

In the following output the "num" column represents the line number within each Chain (INPUT, FORWARD, and OUTPUT). This number will be will be used to remove the rules.

Chain INPUT (policy ACCEPT 31267 packets, 2159K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      206 12472 DROP       all  --  *      *       199.188.70.0/24      0.0.0.0/0
2      313 20047 DROP       all  --  *      *       37.59.125.0/24       0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 14419 packets, 75M bytes)
num   pkts bytes target     prot opt in     out     source               destination

Look at the number on the left in the "num" column, then use number to delete the rule. For example delete line number 2 (subnet 37.59.125.0/24), enter:

sudo iptables -D INPUT 2

This will not take effect until it the rule is saved:

# Save the iptables rules
sudo /etc/init.d/iptables save

Identify Other IP Addresses Performing Similar Attacks

Use the following command to identify IP addresses making large number of login attempts the WordPress blog:

# IP addresses making POST request to the WordPress login page
grep wp-login /var/log/httpd/access_log | grep POST | awk '{print $1}'| sort | uniq -c | sort -nr

  20154 199.188.70.163
  14105 194.65.224.242
   3732 37.59.125.22
     40 37.1.222.114
     25 192.187.99.194

Make sure that you EXCLUDE remove your own IP addresses and other legitimate IP addresses from the above list before blocking them. In this case we should probably block the first three ( 20154 199.188.70.163, 194.65.224.242, and 37.59.125.22) and not the last two (37.1.222.114 and 192.187.99.194).

You may want to use an IP address lookup tool such as MaxMind GeoIP Lookup to see where the IP address is located:

Finding Your Public IP Address on the Command Line

The following is a Bash command that can be used to quickly find the public IP address your internet service provider (ISP) has given you. This is not the same as the IP address seen by your broadband router, which is usually an private IP address within the ISPs network, or the IP address given to your computer, which is a private IP address issued by my home router.

This command simply makes a "what is my ip" query to Google, which in turn displays the IP address it sees the request originating from. The awk command parses the IP address from the rest of the HTML and prints it to the command line.

curl -s 'https://www.google.com/search?q=what+is+my+ip' | awk 'match($0, /\(Client.*: (.*)\)/){print substr($0, RSTART, RLENGTH)}'

(Client IP address: 50.x.x.x)

Using Gnuplot in Bash Shell Scripts

This is an example of how Gnuplot scripts can be embedded within Bash scripts using Bash Here Documents (heredocs):

#!/bin/bash

filename="log/load_me.log.1368725550"

awk '/for 10000000/ {c+=1; if(c > 46){n+=1; printf "%s\t%s\n", n, $2} }' $filename > log/values.dat

window_size=50
ruby running_average.rb log/values.dat $window_size > log/averages.dat

gnuplot -p <<EOSCRIPT

set title 'Time to Insert Relationships in Batches of 10M in Neo4j batch-import'
set xlabel 'Batches of 10M Rels on `date "+%Y-%m-%d at %H:%M:%S %Z"`'
set ylabel 'Time Taken (s)'
set grid

# Draw trend line
f(x) = a*x**3 + b*x**2 + c*x + d

set fit quiet
fit f(x) 'log/values.dat' using 1:(\$2/1000) via a,b,c,d
unset fit

# Scale column No.2 by 1000 to turn ms in s.
plot 'log/values.dat' using 1:(\$2/1000) title 'Time to Insert Rels' with lines, f(x) title 'Fit', 'log/averages.dat' using 1:5 title "Average $window_size" with lines

pause -1 "\n\nHit return to continue\n\n"

EOSCRIPT

num_stats=`wc -l log/values.dat | awk '{print $1}'`

tail -$window_size log/values.dat | awk -v num_stats="$num_stats" '{n+=1; s+=$2}
END{
  avg=s/(n*1000);
  print "Average of last", n, "is", avg, "(s)";
  print "Num Rels Stats =", num_stats;
  print "Hours Remaining =", (3300-num_stats+46)*avg/3600;
  printf "Percentage complete = %.2f%%\n", (num_stats * 100 / 3300.0); 
}'

echo -e "Completed at `date`\n\n"

References

Gnuplot fit command

http://www.manpagez.com/info/gnuplot/gnuplot-4.6.0/gnuplot_263.php

Bash Here Documents

https://www.tldp.org/LDP/abs/html/here-docs.html

Wikipedia Here Document

https://en.wikipedia.org/wiki/Here_document

WPM: How Fast Can You Really Type?

I recently read a job application form that contained a rather interesting question. The question read "Typing Speed (WPM):". Now this would be a perfectly normal question if the job application was for a secretary or typist position, but the job in question was actually for a software engineering position.

I have been touch-typing (yes, all ten fingers--not just the two index fingers) since I was about 10 years old, and I must say that I can type rather quickly. But I have never actually measured my typing speed in terms of words-per-minute (WPM). So I decided to fire up a Linux console and find out once and for all.

The method I used is actually very simple.

1. Type a bunch of words in the console
2. Find out how many words I typed
3. Find out how long it took me to type those words
4. Use those two figures to compute my typing speed in words-per-minute

Here is what I did:

time cat | wc -w
The quick brown fox jumped over the lazy dog's head
[^D]

10

real    0m9.113s
user    0m0.008s
sys     0m0.000s

For those of you who are new to the Linux console, this is what is happening in the above chain of commands:

• The cat command captures whatever you type at the keyboard until you type Ctrl+D.
• This text is then "piped" to the wc (Word Count) command. The -w option tells wc to return the number of words in the text piped to it.
• The above mentioned commands are executed through the time command which measures how much time it took to execute those commands. What is of interest is the first line with the "real"time required to execute.

Computing the Words-per-Minute

The output of the chain of commands (also known as a "pipeline") tells us the two things we need to compute my typing speed: The number of words I typed and the time it took me to type them. Here's some simple math:

In 9.113 seconds I typed 10 words
So in 1 second I could type 10/9.113 words
So in 60 seconds (1 minute) I could type 60*10/9.113

Therefore my typing speed is 65.84 words per minute

Improving the Accuracy

You can only get a rough idea of your typing speed if you type in only 10 words. To get an accurate idea of your typing speed you should try entering a paragraph or two (or more) of text and using the above formula to compute the speed.

Characters-per-Minute

If you want to compute the number of characters you can type in a minute, you should use the wc command's -c switch instead of -w.