Ryan Byrd's Tech Ramblings

1. password
2. 123456
3. qwerty
4. abc123
5. letmein
6. monkey
7. myspace1
8. password1
9. link182
10. (your first name)

If you’ve read On Intelligence, by Jeff Hawkins, you probably remember him saying that people wouldn’t want to build a smart robot that resembled a person. If you’re like me, you probably said to yourself, “Jeff, that’s the dumbest assertion in the world. Of course people will build humanoid robots.” In fact, that’s really the point, right? We need robot soldiers, robot servants, robot workers, etc. Anyway, Jeff, nice book, but you’re WAY OFF.

And now we have robots that can walk, semi-decently, and it’s hard to push them over:

Said one blogger, describing the robot, it’s a “mindless, unfeeling automaton [that] can walk. Soon it will be able to run, and shortly thereafter it will be able to chase you.”

I, for one, welcome our new robot overlords.

Don’t make fun of me for this code. It’s very ugly, I know (just the bottom stuff that I wrote, that is). But I wanted to pull down SMS messages from my Google Voice Account last night, so I quickly coded this (with the help of Chad Smith.)

<?php
define('GOOGLE_ACCOUNT','YOU@gmail.com');
define('GOOGLE_PASSWORD','YOURPASSWORDGOESHERE');
function query($data=array(),$fields=array()){
       foreach($data as $k=>$v)
               $fields[]=$k.'='.urlencode(stripslashes($v));
       return implode('&',$fields);
}
function googleapi($url,$post=null,$headers=null){
       $ch=curl_init();
       curl_setopt($ch,CURLOPT_URL,$url);
       if(is_array($post)){
               curl_setopt($ch,CURLOPT_POST,true);
               curl_setopt($ch,CURLOPT_POSTFIELDS,query($post));
       }
       if(is_array($headers))
               curl_setopt($ch,CURLOPT_HTTPHEADER,$headers);
       curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
       curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,2);
       curl_setopt($ch,CURLOPT_SSL_VERIFYPEER,false);
       $data=curl_exec($ch) or curl_error($ch);
       curl_close($ch);
       return $data;
}
$auth=googleapi('https://www.google.com/accounts/ClientLogin',array(
                                               'accountType'=>'GOOGLE',
                                               'Email'=>GOOGLE_ACCOUNT,
                                               'Passwd'=>GOOGLE_PASSWORD,
                                               'service'=>'grandcentral',
                                               'source'=>'googlevoice.php'));
if(preg_match('/Error=([A-z]+)/',$auth,$error))
       die($error[1]);
if(preg_match('/Auth=([A-z0-9_-]+)/',$auth,$auth));
       $auth=$auth[1];
preg_match("/'_rnr_se'\: '([^']+)'/",googleapi('https://www.google.com/voice/?auth='.$auth),$rnr);
$rnr=$rnr[1];
function sms($them,$text){
       global $auth,$rnr;
       return googleapi('https://www.google.com/voice/sms/send/?auth='.$auth,array(
               'id'=>'',
               'phoneNumber'=>$them,
               'text'=>$text,
               '_rnr_se'=>$rnr)
       );
}
 
function get_xml($url){
       global $auth;
       return googleapi($url.'?auth='.$auth);
}
 
$sms_list=get_xml('https://www.google.com/voice/inbox/recent/sms/');
$sms_list=str_replace("\n"," ",$sms_list);
$sms_list=str_replace("\t",'',$sms_list);
$sms_list=str_replace("\r",'',$sms_list);
$sms_list=str_replace("  "," ",$sms_list);
$pattern='/<table class="gc-message-transcript-middle">.*?<div class="gc-message-sms-row">(.*?<\/span>) <\/div>.*?<\/table>/';
$matches=array();
preg_match_all($pattern, $sms_list, $matches);
for($x=0; $x<count($matches[1]); $x++)
{
       do
       {
               $found=0;
               list($from, $text, $time)=split('</span>',$matches[1][$x]);
               $from=trim(strip_tags($from));
               $text=trim(strip_tags($text));
               $time=trim(strip_tags($time));
               if (substr($text,0,2)=="+1")
               {
                       $pos=strpos($text,' - '); //find first (leftmost) occurrence of hyphen
                        if ($pos !== false)
                        {
                                $from=substr($text,0,$pos);
                                $text=substr($text,$pos+3);
                        }
                       $from=substr(trim(strip_tags($from)),2);
                       $text=trim(strip_tags($text));
               }
               $from=str_replace(array('(',')','-',' ',':'),'',$from);
               if (is_numeric($from[0]))
               {
                       echo("\n from: $from, text: $text, time: $time\n");
                       $matches[0][$x]=str_replace($matches[1][$x],'',$matches[0][$x]);
                       $multitext=array();
                       preg_match_all($pattern, $matches[0][$x],$multitext);
                       if (count($multitext[1])>0)
                       {
                               $matches[1][$x]=$multitext[1][0];
                               $found=1;
                       }
                       if (trim($matches[1][$x])=='') $found=0;
               }
       } while($found);
}
?>

If you import the ip2location table into mysql you get a 2.6 million row table:

mysql> select count(*) from ip_lookup;
+----------+
| count(*) |
+----------+
|  2969801 |
+----------+
1 row in set (0.00 sec)
 
mysql>

If you try to query that table (even after adding indexes on fromip and toip), it takes a couple of seconds:

[root@localhost ~]# time php test.php
ALPINE,UTAH,UT
real    0m2.642s
user    0m0.012s
sys     0m0.008s
[root@localhost ~]#

test.php is running this query:

select * from ip_lookup where toip>='1077033402' and fromip<='1077033402'

I created a new table with an autoincrement primary key and imported the data from ip_lookup into it:

CREATE TABLE ip_lookupid (id INT AUTO_INCREMENT PRIMARY KEY)
SELECT * FROM ip_lookup order by fromip;

then I created an ip_lookup_master table:

create table ip_lookup_master (  `fromip` int(10) unsigned zerofill NOT NULL default '0000000000',
  `toip` int(10) unsigned zerofill NOT NULL default '0000000000', tablename varchar(50));

I then used excel’s awesome concatenate function, combined with some simple formulas to carve up the big table and create SQL statements like:

CREATE TABLE ip_lookup1 SELECT * from ip_lookupid where id between 2821311 and 2969801;
INSERT INTO ip_lookup_master (fromip,toip,tablename) values((select min(fromip) from ip_lookup1), (select max(toip) from ip_lookup1), 'ip_lookup1');

that produced a lookup table like this:

mysql> select * from ip_lookup_master;
 +------------+------------+-------------+
 | fromip     | toip       | tablename   |
 +------------+------------+-------------+
 | 3630847424 | 4294967295 | ip_lookup1  |
 | 3536289792 | 3630847431 | ip_lookup2  |
 | 3492514992 | 3536293759 | ip_lookup3  |
 | 3409495296 | 3492515583 | ip_lookup4  |
 | 3267741056 | 3409495551 | ip_lookup5  |
 | 2903513088 | 3267741183 | ip_lookup6  |
 | 2063107016 | 2903513599 | ip_lookup7  |
 | 1565740420 | 2063107023 | ip_lookup8  |
 | 1459814656 | 1565740447 | ip_lookup9  |
 | 1354800128 | 1459816191 | ip_lookup10 |
 | 1267933104 | 1354800383 | ip_lookup11 |
 | 1211338496 | 1267933111 | ip_lookup12 |
 | 1176006656 | 1211339007 | ip_lookup13 |
 | 1146034944 | 1176007679 | ip_lookup14 |
 | 1117799264 | 1146035199 | ip_lookup15 |
 | 1097247648 | 1117799271 | ip_lookup16 |
 | 1075237704 | 1097247655 | ip_lookup17 |
 | 1048990720 | 1075237727 | ip_lookup18 |
 | 0216716528 | 1048991743 | ip_lookup19 |
 | 0000000000 | 0216716535 | ip_lookup20 |
 +------------+------------+-------------+
 20 rows in set (0.00 sec)
 
 mysql>

Then I added another query to my php lookup script:

select tablename from ip_lookup_master where '1077033402'>fromip and '1077033402'<toip;
select * from ip_lookup17 where toip>='1077033402' and fromip<='1077033402';

and now things are a LITTLE bit faster:

[root@localhost ~]# time php test.php
ALPINE,UTAH,UT
real    0m0.031s
user    0m0.012s
sys     0m0.016s
[root@localhost ~]#

Local Traffic-> SSL Certificates
Click Import…
Select Key
name key, upload key file
select the key you just created, then click on Import…
upload the corresponding cert
Click Import…
Select Cert
upload your intermediate cert (if applicable)
Local Traffic-> Profiles
mouse over SSL, select client
click Create…
name the profile, click custom, select Advanced Configuration
select your cert and key
under trusted certificate authories, select your intermediate cert (if applicable)
Click Finished
Local Traffic-> Nodes
Create, name, ip address, real_server monitor
Local Traffic->Pools
Create, add your nodes, http monitor
Local Traffic->Virtual Servers
Create, name, port, http profile, ssl profile (that you just created), select the default pool

if you want an automagic 80-443 redirect, create an irule, then paste this in:

when HTTP_REQUEST {
HTTP::redirect https://[HTTP::host][HTTP::uri]
}

then associate the irule with a port 80 server. If you want to preserve the remote address, attach this irule

when HTTP_REQUEST {
      HTTP::header insert X-Forwarded-For [IP::remote_addr]
      }

Imagine you have a MySQL server, db1, behind a firewall. You have another machine, lb1, that is behind the firewall as well, but it has port 22(SSH) forwarding turned on. You want your local development server, dev1, to be able to connect to db1. How do you do that?
Tunnel from db1 to lb1

ssh -N -f -R 3306:127.0.0.1:3306 lb1 -l root

Tunnel from dev1 to lb1

ssh -L 3333:127.0.0.1:3306 lb1 -l root

or

/usr/bin/autossh -M 9877 -c blowfish -CN -L 3333:127.0.0.1:3306 root@lb1 &

and then you should be able to connect to lb1 from dev1 like this:

mysql -u root --host=127.0.0.1 --port=3333

If that doesn’t work, add these lines in your /etc/ssh/sshd_config file and restart the sshd service:

AllowTcpForwarding yes
GatewayPorts       yes

For a couple hundred dollars you can get an ip2location database with IP address to city and state correlation. First you’ll need to load the data they give you by creating a MySQL database and table. Then load the CSV into that database table. The table SQL code:

CREATE TABLE `ip_lookup` (
`fromip` int(10) unsigned zerofill NOT NULL default '0000000000',
`toip` int(10) unsigned zerofill NOT NULL default '0000000000',
`countrycode` char(2) NOT NULL default '',
`countryname` varchar(64) NOT NULL default '',
`region` varchar(128) NOT NULL default '',
`city` varchar(128) NOT NULL default '',
PRIMARY KEY  (`fromip`,`toip`) )
ENGINE=MyISAM DEFAULT CHARSET=latin1;

and the import:

LOAD DATA INFILE '/usr/local/src/IP-COUNTRY-REGION-CITY.CSV'
     INTO TABLE ip_lookup
     FIELDS TERMINATED BY ","
     OPTIONALLY ENCLOSED BY """"
     LINES TERMINATED BY "\r\n";

To find the city and state, convert the IP dotted decimal to a number, and then lookup in the database. Here’s some quick code that will give you a nifty HTML text box and then return the geoip info in a chart or allow you to download as a CSV

function IPAddress2IPNumber($dotted)
{
        $dotted = preg_split("/[.]+/",$dotted);
        $ip = (double) ($dotted[0] * 16777216) + ($dotted[1] * 65536) + ($dotted[2] * 256) + ($dotted[3]);
        return $ip;
}
$States = array('AL'=>"Alabama",  'AK'=>"Alaska",  'AZ'=>"Arizona",  'AR'=>"Arkansas",  'CA'=>"California",  'CO'=>"Colorado",  'CT'=>"Connecticut",  'DE'=>"Delaware",  'DC'=>"District Of Columbia",  'FL'=>"Florida",  'GA'=>"Georgia",  'HI'=>"Hawaii",  'ID'=>"Idaho",  'IL'=>"Illinois",  'IN'=>"Indiana",  'IA'=>"Iowa",  'KS'=>"Kansas",  'KY'=>"Kentucky",  'LA'=>"Louisiana",  'ME'=>"Maine",  'MD'=>"Maryland",  'MA'=>"Massachusetts",  'MI'=>"Michigan",  'MN'=>"Minnesota",  'MS'=>"Mississippi",  'MO'=>"Missouri",  'MT'=>"Montana",'NE'=>"Nebraska",'NV'=>"Nevada",'NH'=>"New Hampshire",'NJ'=>"New Jersey",'NM'=>"New Mexico",'NY'=>"New York",'NC'=>"North Carolina",'ND'=>"North Dakota",'OH'=>"Ohio",  'OK'=>"Oklahoma",  'OR'=>"Oregon",  'PA'=>"Pennsylvania",  'RI'=>"Rhode Island",  'SC'=>"South Carolina",  'SD'=>"South Dakota",'TN'=>"Tennessee",  'TX'=>"Texas",  'UT'=>"Utah",  'VT'=>"Vermont",  'VA'=>"Virginia",  'WA'=>"Washington",  'WV'=>"West Virginia",  'WI'=>"Wisconsin",  'WY'=>"Wyoming");
$invertStates=array_flip($States);
foreach($invertStates as $state=>$abbrev)
        $invertStates[strtoupper ($state)]=$abbrev;
if (isset($_REQUEST['ipaddresses']))
{
        $ips=str_replace("\n",',',$_REQUEST['ipaddresses']);
        $ips=split(',',$ips);
        array_walk($ips,'trim');
        foreach($ips as $ip)
        {
                $ip=trim($ip);
                if ($ip=='') continue;
                $number=IPAddress2IPNumber($ip);
                mysql_connect('localhost', 'username', 'password') or die(mysql_error());
                mysql_select_db('database') or die(mysql_error());
                $iptable="ip_lookup";
                $sql="select * from ".$iptable." where toip>='".$number."' and fromip<='".$number."' limit 1";
                $result=mysql_query($sql) or die(mysql_error());
                if ($result)
                {
                         $set=mysql_fetch_array($result);
                         $city=$set['city'];
                         $region=$set['region'];
                         if ($city!='-')
                        {
                                 $geoArray[][$ip]="$city, $region";
                        }
                        else
                        {
                                $geoArray[][$ip]="not found";
                        }
 
                }
        } //foreach
        if (isset($_REQUEST['submit']))
        {
                echo("<html><head></head><body>");
                echo("<table cellpadding=\"5\" border=\"1\">");
                foreach($geoArray as $subArray)
                {
                        foreach($subArray as $ip=>$locale)
                        {
                                list($city,$state)=split(',',$locale);
                                $state=trim($state);
                                echo("<tr><td>$ip</td><td>$locale</td><td>$invertStates[$state]</td></tr>");
                        }
                }
                echo("</table></body></html>");
        }
        else
        {
                header("Expires: 0");
                header("Cache-control: private");
                header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
                header("Content-Description: File Transfer");
                header("Content-Type: application/vnd.ms-excel");
                header("Content-disposition: attachment; filename=".date("Y-m-d").".csv");
                echo('"IP Address","City and State","State Abbreviation"');
                echo("\015\012");
 
		foreach($geoArray as $subArray)
                {
                        foreach($subArray as $ip=>$locale)
                        {
                                list($city,$state)=split(',',$locale);
                                $state=trim($state);
                                echo('"'.$ip.'","'.$locale.'","'.$invertStates[$state].'"'."\015\012");
                        }
                }
                exit;
        }
}
?>
<h1>Enter IP addresses, one per line, or separated by commas</h1>
<form action="<?echo($PHP_SELF);?>" method="POST">
<textarea name="ipaddresses" cols="40" rows="10"><?echo($_REQUEST['ipaddresses']);?></textarea>
<input type="submit" name="submit" value="geolookup ips" />
<input type="submit" name="csv_submit" value="download as CSV" />
</form>

If that’s too slow for you, you could always segment the BIG ip_lookup table into smaller tables. Then create a master table called iptable that looks-a-like this:

+-----------+------------------+------+-----+---------+----------------+
| Field     | Type             | Null | Key | Default | Extra          |
+-----------+------------------+------+-----+---------+----------------+
| iptableid | int(11)          | NO   | PRI | NULL    | auto_increment |
| tablename | varchar(50)      | NO   | MUL |         |                |
| ip        | int(10) unsigned | NO   | MUL | 0       |                |
+-----------+------------------+------+-----+---------+----------------+

That table lets us know which of the other tables to look in. The child tables have the following structure:

+-------------+---------------------------+------+-----+------------+-------+
| Field       | Type                      | Null | Key | Default    | Extra |
+-------------+---------------------------+------+-----+------------+-------+
| fromip      | int(10) unsigned zerofill | NO   | PRI | 0000000000 |       |
| toip        | int(10) unsigned zerofill | NO   | PRI | 0000000000 |       |
| countrycode | char(2)                   | NO   |     |            |       |
| countryname | varchar(64)               | NO   |     |            |       |
| region      | varchar(128)              | NO   |     |            |       |
| city        | varchar(128)              | NO   |     |            |       |
+-------------+---------------------------+------+-----+------------+-------+

Before you write and tell me that there are alternatives to ip2location like maxmind and hostip.info, you should know that, according to my tests, ip2location is more accurate and, because you get to download the entire database to your own server, you can ensure it’s always up and make it fast.

Maybe you’re getting credit card info that you have to save to disk. Maybe at the end of each day you ftp those credit cards to another server to batch process. Maybe you’d like to encrypt that info. What do you do?

Encryption process:
STEP 1: generate public and private keys with gnupg (gpg –gen-key), then move the private key(~/.gnupg/secring.gpg) to a different server
STEP 2: serialize the array of order info, then base64 encode it
STEP 3: use the public key to encrypt that

Decryption process:
STEP 4: use the private key to unencrypt, then base64 unencode, then unserialize
How ’bout some code for encryption:

$to = "myemail@domain.com";
$order=array("First Name"=>"Jason","Last Name"=>"Bourne","Credit Card"=>"4111111111111111","CCV"=>"220","Favorite Ice Cream"=>"Chocolate","Transaction ID"=>"23478923478942398");
$serialOrder=base64_encode(serialize($order));
print($serialOrder);
$gpg_path = '/usr/bin/gpg';
$home_dir = '/root';
$user_env = 'apache';
$cmd = "echo $serialOrder | HOME=$home_dir USER=$user_env $gpg_path" .
  " --no-secmem-warning --encrypt ".
  " --recipient $to";
  $encrypted = `$cmd`;
$fp = fopen('encrypted.txt', 'w');
fwrite($fp, $encrypted);
fclose($fp);

and for decryption:

$cmd=`/usr/bin/gpg --output unencrypted.txt --decrypt encrypted.txt`;
$unencrypted=file_get_contents("unencrypted.txt");
print_r(unserialize(base64_decode($unencrypted)));

references:

I’m not talking about the chemical element Selenium (atomic number 34) discovered by J.J. Berzelius, a Swede, in 1817. I’m talking instead about an open source web application testing system. Basically, you record your interactions with a web app and then you can play that script back, modifying it if you choose to add assertions, etc. And you can play that script back in different browsers on different platforms. That’s what I set up this week. Here’s how I did it.

Computer 1: the test master
This windows XP PC has XAMPP, and the Selenium php libraries (which also required PHP PEAR phpunit to be installed.) Additionally, I installed cygwin with openssh for the 2 forward ssh tunnels that run to the Mac. On this computer are the recorded selenium scripts.
Computer 2: the test target
This Mac Pro is a beefy box that is running two virtualbox instances of Windows XP. Additionally, it has Firefox and Safari browsers installed and the selenium server (a java program) installed. the networks to the VMs are natted (bridging wouldn’t work for some reason)
VirtualBox Windows XP instance 1
this XP VM has IE 6 and firefox browsers. It also has cygwin w/ openssh and a reverse tunnel set up to the Mac. It is running the selenium server.
VirtualBox Windows XP instance 2
this XP VM has IE 7and chrome browsers installed. Like the other VM, it has cygwin w/ openssh and a reverse tunnel set up to the Mac. It is also running the selenium server

Out-of-the-box selenium doesn’t have an time-limit assert. I modified phpunit to allow that. There is a PHPUnit_Extensions_SeleniumTestCase which extends PHPUnit_Framework_TestCase. There is another class, PHPUnit_Extensions_PerformanceTestCase that has some timing functionality that I put into PHPUnit_Extensions_SeleniumTestCase because the classes we record and run can’t extend two classes– no multiple inheritance in PHP. Inside the abstract class PHPUnit_Extensions_PerformanceTestCase I added:

$protected $maxRunningTime=0;

and then inside the method runTest() I added, after $this->start;

PHPUnit_Util_Timer::Start();

and then at the end of that method, I put:

$time = PHPUnit_Util_Timer::stop();
 
       if ($this->maxRunningTime != 0 &&
           $time > $this->maxRunningTime) {
           $this->fail(
             sprintf(
               'expected running time: <= %s but was: %s',
 
               $this->maxRunningTime,
               $time
             )
           );
       }

Then, at the start of the recorded selenium script, for a twenty second timer, add:

$this->setMaxRunningTime(20);

Now for the details of the forward and reverse ssh tunnels and the code to connect to the target Mac box:

on win xp vm 1
	ssh -R 4133:127.0.0.1:4444 macIpAddr -l username
 
on win xp wm2
	ssh -R 4134:127.0.0.1:4444 macIpAddr -l username
 
on the test runner box
	ssh -L 4133:127.0.0.1:4133 macIpAddr -l username
	ssh -L 4134:127.0.0.1:4134 macIpAddr -l username
 
to open win xp vm 1
	$this->setHost('127.0.0.1');
	$this->setPort('4133');
 
to open win xp vm 2
	$this->setHost('127.0.0.1');
	$this->setPort('4134');

Today we tried to make a SOAP request to a .net server, but it kept complaining “Object reference not set to an instance of an object”. Annoying. After 1/2 hour of getting nowhere, i simply used curl to pass the xml request, a-like-a-this:

<?php
 
$xmlRequest = <<<XYZZ
<?xml version="1.0" encoding="UTF-8" standalone="no"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tm="http://microsoft.com/wsdl/mime/textMatching/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:tns="http://tempuri.org/" xmlns:s="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://schemas.xmlsoap.org/wsdl/soap12/" xmlns:http="http://schemas.xmlsoap.org/wsdl/http/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" ><SOAP-ENV:Body><tns:InsertNewUser xmlns:tns="http://tempuri.org/"><tns:argUserContainer><tns:FirstName>first</tns:FirstName><tns:LastName>last</tns:LastName><tns:UserLogin>login</tns:UserLogin><tns:UserPassword>pass</tns:UserPassword><tns:StartedDate>2009-01-01</tns:StartedDate><tns:LastVisitDate>2009-01-01</tns:LastVisitDate><tns:IsActive>1</tns:IsActive><tns:PersistentID>1</tns:PersistentID></tns:argUserContainer></tns:InsertNewUser></SOAP-ENV:Body></SOAP-ENV:Envelope>
XYZZ;
 
 
$header[] = "Content-type: text/xml";
$header[] = "Content-length: ".strlen($xmlRequest) . "\r\n";
$header[] = $xmlRequest;
 
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://IPADDRESSREMOVED/membershipservice.asmx?WSDL");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt( $ch, CURLOPT_HTTPHEADER, $header );
curl_setopt( $ch, CURLOPT_CUSTOMREQUEST, 'POST' );
$response = curl_exec($ch);
curl_close($ch);
echo htmlentities($response);
exit;
 
?>

I used this site to generate the XML from the WDSL: http://www.soapclient.com/soaptest.html