Ryan Byrd's Tech Ramblings

Remote Terminal Data Pushes (okay, okay– polling)

me — Tue, 31 Aug 2010 19:34:26 +0000

Maybe you have a giant warehouse and you’d like some large LCD monitors around the shipping lines which have the orders listed for all to see. You’d like these LCDs to update frequently and to be specific to each shipping line.

What to do? Well, there is meta-refresh every second, but that’s flickery and lame. Let’s get the javascript prototype library to update divs based on the location selected in a dropdown list:


<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>LCD Terminal Somewhere</title>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/prototype/1.6.0.2/prototype.js"></script>
<body>
<script>
                var url = 'get_content.php';
                var pars = 'location=1';
                var myAjax = new Ajax.PeriodicalUpdater(
                        'placeholder',
                        url,
                        {
                                method: 'post',  //use post or IE will cache(with get)
                                parameters: pars,
                                frequency: 2
                        });
</script>
</head>
<body>
<h2>Terminal Display</h2><i>(content refreshes every two seconds)</i><br />
   <div id="placeholder" style="width:600px; height: 300px; border-width: .1em;border-style: solid;border-color: #000;">Lorum Ipsum</div>
<p><p><p>
<form>
<label for="myLocation">Change Location: </label><select name="myLocation" id="myLocation" onchange="pars='location='+this.options[this.selectedIndex].value; myAjax.options.parameters = pars.toQueryParams();">
<option value="1">Line 1</option>
<option value="2">Lines 2 and 3</option>
<option value="3">Reception</option>
<option value="4">Break Room</option>
</select>
</form>
 
</body>
</html>

And here is get_content.php


switch($_REQUEST['location'])
{
        case 1:
	        echo("Some custom data for Line 1
");
	        break;
        case 2:
	        echo("Custom datagrid for Lines 2 and 3
");
	        break;
        case 3:
	        echo("This would be displayed on the receptionist LCD
");
	        break;
        case 4:
	        echo("Content for the break room
");
	        break;
}
print("
\"right\">Last Updated: ");
print strftime('%c');
print("
");

PHP ping and DNS lookup

me — Wed, 25 Aug 2010 23:02:08 +0000

here’s a handy utility script so you can check pings and DNS from a remote server though a web browser




IP address
or
hostname



function ping($host) {
    exec(sprintf('ping -c 1 -W 5 %s', escapeshellarg($host)), $res, $rval);
    return $res;
  }
 
if (isset($_REQUEST['ipaddr'])&&$_REQUEST['ipaddr']!='')
{
        if(preg_match("/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/",$_REQUEST['ipaddr']))
        {
                echo("");print_r(ping($_REQUEST['ipaddr']));echo("
");
        }
        else { echo("ip address not valid");}
}
if (isset($_REQUEST['host'])&&$_REQUEST['host']!='')
{
        if (preg_match('/^([a-z0-9][a-z0-9-]{0,62}\.)+([a-z]{2,4})$/',$_REQUEST['host']))
        {
                $result = dns_get_record($_REQUEST['host']);
                echo("");print_r($result);echo("
");
        }
        else { echo("host name not valid");}
}
?>

BASH script to check URL for changes

me — Mon, 16 Aug 2010 20:00:14 +0000

Here’s a quick script I coded this morning to check a URL/URLs for changes and alert you

#!/bin/bash
HASHFILE="hashes.txt"
URLFILE="urls.txt"
EMAILADDRESS="name@example.com"
if [ -z "$1" ]
then
        echo "Usage: `basename $0` [TRAIN|CHECK]"
        echo "Create a file called urls.txt with URLS to check"
        exit 1
fi
if [[ "${1}" == "CHECK" ]]
then
        echo "CHECKING"
        if [ -f $HASHFILE ]
        then
                echo "Hash file exists. Parsing"
                while read line; do
                        URL=`echo $line |awk '{print $1}'`
                        HASH=`echo $line |awk '{print $2}'`
                        echo "Checking $URL"
                        wget $URL >& /dev/null
                        FILENAME=`basename $URL`
                        HASHVALUE=`md5sum $FILENAME|awk '{print $1}'`
                        if [[ "${HASHVALUE}" == "${HASH}" ]]; then
                                echo "NOTICE: hasn't changed"
                        else
                                echo "ALERT: [$URL] CHANGED from [$HASH] to [$HASHVALUE]"
                                PREFIX="BLOCK_"
                                BLOCKFILE=$PREFIX$FILENAME
                                if [ -f $BLOCKFILE ]
                                then
                                        echo "BLOCK FILE EXISTS. NOT EMAILING"
                                else
                                        echo "NO BLOCK FILE, EMAILING ALERT. WRITING BLOCK FILE"
                                        mail -s "ALERT: [$URL] CHANGED from [$HASH] to [$HASHVALUE]" $EMAILADDRESS dev/null
                                        echo "$URL changed. alerted" > $BLOCKFILE
                                fi
                        fi
                        rm -f $FILENAME
                done < $HASHFILE
        else
                echo "ERROR: no hash file exists. Exiting"
                exit 1
        fi
else
        echo "TRAINING"
        if [ -f $URLFILE ]
        then
                echo "URL file exists. Parsing"
                rm -f $HASHFILE
                touch $HASHFILE
                while read line; do
                        wget $line >& /dev/null
                        FILENAME=`basename $line`
                        HASHVALUE=`md5sum $FILENAME|awk '{print $1}'`
                        echo "$line $HASHVALUE" >> $HASHFILE
                        echo "added $line to hash file"
                          rm -f $FILENAME
                done < $URLFILE
        else
                echo "ERROR: no url file exists. Exiting"
                exit 1
        fi
fi

Date Interval Calculation Function

me — Wed, 04 Aug 2010 05:12:24 +0000

You’d like to email someone seven times, every other day starting today. How do you get those dates?

function calculateIntervalDates($startDate, $interval, $count)
{
        $intervalArray=array('Daily'=>1,'Every-other-day'=>2,'Weekly'=>7, 'Twice-monthly'=>15, 'Monthly'=>30);
        $daysToAdd=$intervalArray[$interval];
        for($x=0;$x<$count;$x++)
        {
                $intervalDateArray[]= date("m/d/Y", strtotime("$startDate + ".$daysToAdd*$x." days"));
        }
        return $intervalDateArray;
}
 
print_r(calculateIntervalDates("08/03/2010",'Every-other-day',7));

which outputs

Array
(
    [0] => 08/03/2010
    [1] => 08/05/2010
    [2] => 08/07/2010
    [3] => 08/09/2010
    [4] => 08/11/2010
    [5] => 08/13/2010
    [6] => 08/15/2010
)

I recognize the monthly is an approximate and that I could do better by having a special case for that, but, it’s close enough.

Setting User-Agent Header in Selenium

me — Mon, 02 Aug 2010 18:40:18 +0000

If you Google “setting the Selenium User-Agent header” you get a whole bunch of crapola. Specifically, Selenium folks want you to use web drivers, which is stupid because the whole point is that you want to be able to record tests in the Selenium Firefox plugin and then run them with Selenium RC. (setting the user-agent is useful when trying to filter out tests from live traffic)

Anyway, the solution is to create a Firefox profile with the user-agent string already set. Be sure to modify the profile to clear cookies and cache on exit, to not prompt for updates or add-ons etc.

First, start Firefox in profile editor mode:

firefox.exe -ProfileManager -no-remote

Go ahead and create a new profile and then start Firefox with that profile and make you customizations (set the user-agent, etc). Then launch selenium like this:

java -jar selenium-server.jar -firefoxProfileTemplate "your profile directory goes here"

Here is some test code to see if you’ve succeeded:

require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
 
class testuagent extends PHPUnit_Extensions_SeleniumTestCase
{
  function setUp()
  {
    $this->setBrowser("*firefox");
    $this->setBrowserUrl("http://www.iopus.com/");
  }
 
  public function testMyTestCase()
  {
 
    $this->open("/imacros/demo/v5/user-agent.htm");
    $this->waitForPageToLoad("30000");
    sleep(20);
  }
}

useful links:

http://johnbokma.com/mexit/2004/04/24/changinguseragent.html

http://girliemangalo.wordpress.com/2009/02/05/creating-firefox-profile-for-your-selenium-rc-tests/

http://www.iopus.com/imacros/demo/v5/user-agent.htm

http://support.mozilla.com/en-US/kb/How+to+clear+the+cache

Hip, Hip Array

me — Sun, 01 Aug 2010 05:40:50 +0000

I couldn’t resist…

src: http://www.neatoshop.com/product/Hip-Hip-Array

Selenium, Nagios and Magento– together at last!

me — Thu, 29 Jul 2010 16:51:26 +0000

Now that I have a server running selenium tests against a magento-powered ecommerce site and logging those tests to a file, we need to integrate with nagios. First, I installed openSSH on that windows server (via copssh), and copied over the nagios user public key to authorized_keys. Then I wrote this nagios script:

#! /usr/bin/php -q

ini_set( 'max_execution_time', '55' );
ini_set( 'display_errors', false );
 
define( "STATUS_OK",       0 );
define( "STATUS_WARNING",  1 );
define( "STATUS_CRITICAL", 2 );
define( "STATUS_UNKNOWN",  3 );
 
$execCommand='ssh WINDOWS_SERVER_IP_GOES_HERE -l nagiosremote "tail -n 1 selenium.txt"';
exec($execCommand,$output);
list($id,$date,$time,$status,$avg,$std,$FLAG,$flagCnt,$statCnt)=split(',',$output[0],9);
if ((strpos($status,'OK')===false)||(strpos($FLAG,'ERROR')!==false))
{
        echo("ERROR: $status $FLAG\n");
        exit( STATUS_CRITICAL );
}
else
{
        echo("OK: time $time, status $status, avg $avg, std $std, FLAG $FLAG, flagCnt $flagCnt, statCnt $statCnt\n");
        exit( STATUS_OK );
}

Then I defined a check_command, a host, a host_group and a service in nagios and we’re in business!

Quick Selenium RC Runner Script (updated)

me — Tue, 27 Jul 2010 22:31:10 +0000

define('BADWEIGHT',10);
define('GOODWEIGHT',1);
define('BADMULTIPLIER',150);
define('BADTIMEOUT',10);
define('SLOWDEVIATIONS',2);
$filename = 'C:\Program Files\ICW\home\nagiosremote\selenium.txt';
 
$execCommand="phpunit.bat selenium.php";
$trial=0;
function logToFile($somecontent)
{
	global $filename;
	if (is_writable($filename))
	{
		if (!$handle = fopen($filename, 'a'))
		{
			 echo "Cannot open file ($filename)";
			 exit;
		}
		if (fwrite($handle, $somecontent) === FALSE)
		{
			echo "Cannot write to file ($filename)";
			exit;
		}
		fclose($handle);
	}
	else
	{
		echo "The file $filename is not writable";
	}
}
 
function sd_square($x, $mean) { return pow($x - $mean,2); }
 
function sd($array)
{
   return sqrt(array_sum(array_map("sd_square", $array, array_fill(0,count($array), (array_sum($array) / count($array)) ) ) ) / (count($array)-1) );
}
 
function getStats($trials)
{
	foreach($trials as $trial)
	{
		$sum+=$trial['time'];
		$timeArray[]=$trial['time'];
	}
	$counter=(count($trials)>0)?count($trials):1;
	$average=sprintf("%.2f",$sum/count($trials));
	if (count($trials)>1)
	{
		$stdDev=sprintf("%.2f",sd($timeArray));
	}
	else
	{
		$stdDev=0.00;
	}
	return array($average,$stdDev);
}
 
function stripOutliers($times)
{
	$noOutlierArray=array();
	list($avg,$stDev)=getStats($times);
	foreach($times as $lineItem)
	{
		if ($lineItem['time']<=($avg+$stDev))
		{
			$noOutlierArray[]['time']=$lineItem['time'];
		}
	}
	return $noOutlierArray;
}
$flagCnt=0;
$statCnt=0;
while(1)
{
	$output=array();
	exec($execCommand,$output);
	$time=intval(trim(substr($output[4],6,3))); //parse seconds
	list($status,$hud)=split(' ',$output[6],2); //parse status
	$status=trim($status);
	if ($status!='OK')
	{
		$status=implode("-",array($output[6],$output[8],$output[9],$output[10]));
		$statCnt+=BADWEIGHT;
	}
	else
	{
		$statCnt-=GOODWEIGHT;
	}
	$resultArray[$trial]['time']=$time;
	$resultArray[$trial]['status']=$status;
 
	list($avg,$stDev)=getStats(stripOutliers($resultArray));
	$FLAG=($time>($avg+SLOWDEVIATIONS*$stDev))?"SLOW":'OK';
	if ($status!='OK') $FLAG='WARNING! Bad Status ';
	if ($FLAG=='SLOW')
	{
		$flagCnt+=BADWEIGHT;
	}
	else
	{
		$flagCnt-=GOODWEIGHT;
	}
	if ($flagCnt<=0) $flagCnt=0;
	if ($statCnt<=0) $statCnt=0;
	if ($flagCnt>(BADWEIGHT*BADMULTIPLIER)) $FLAG='ERROR! TOO MANY SLOW TRANSACTIONS';
	if ($statCnt>(BADWEIGHT*BADMULTIPLIER)) $FLAG='ERROR! TOO MANY TRANSACTION FAILURES';
	if ($FLAG=='ERROR! TOO MANY TRANSACTION FAILURES'&&$status=='OK') {$FLAG='OK';$flagCnt=0;}
	if ($FLAG=='ERROR! TOO MANY SLOW TRANSACTIONS'&&$status=='OK') {$FLAG='OK';$flagCnt=0;}
	$lineItem="${trial},".date('d/m/Y H:i:s').",$time sec,$status,Avg:$avg,StdDev:$stDev,$FLAG,$flagCnt,$statCnt\n";
	echo($lineItem);
	logToFile($lineItem);
	if (($FLAG!='OK')||($status!='OK')) sleep(BADTIMEOUT);
	$trial++;
}

which will produce a text file like this:

C:\xampp\php>php runTests.php
0,29/07/2010 12:06:01,34 sec,OK,Avg:34.00,StdDev:0,OK,0,0
1,29/07/2010 12:06:20,19 sec,OK,Avg:26.50,StdDev:10.61,OK,0,0
2,29/07/2010 12:06:39,19 sec,OK,Avg:19.00,StdDev:0.00,OK,0,0
3,29/07/2010 12:06:58,18 sec,OK,Avg:18.67,StdDev:0.58,OK,0,0
4,29/07/2010 12:07:17,18 sec,OK,Avg:18.50,StdDev:0.58,OK,0,0
5,29/07/2010 12:07:36,19 sec,OK,Avg:18.60,StdDev:0.55,OK,0,0
6,29/07/2010 12:07:55,18 sec,OK,Avg:18.50,StdDev:0.55,OK,0,0
7,29/07/2010 12:08:14,18 sec,OK,Avg:18.43,StdDev:0.53,OK,0,0
8,29/07/2010 12:08:38,24 sec,OK,Avg:19.12,StdDev:2.03,SLOW,10,0
9,29/07/2010 12:09:12,24 sec,OK,Avg:19.67,StdDev:2.50,OK,9,0

Testing Magento Sites with Selenium

me — Wed, 21 Jul 2010 19:08:29 +0000

So you installed the Selenium IDE firefox plugin and you recorded a check-out transaction of your Magento-powered ecommerce site. Then you save as a php file and fire up your Selenium RC server and run the script. It gets to the checkout page and … Nothing. What’s wrong?

What’s wrong is that you’ve submitted too quickly. You need to do some checking with xpath ids for the “updating totals” div before you can click submit. Here’s a working Selenium Magento script for your reference:

require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
 
class Example extends PHPUnit_Extensions_SeleniumTestCase
{
 function setUp()
 {
   $this->setBrowser("*chrome");
   $this->setBrowserUrl("http://www.example.com");
 }
 
 public function testMyTestCase()
 {
   //$this->maxRunningTime= 30; //(this is a special function I implemented)
   $this->open("/");
   $this->click("//ul[@id='nav']/li[1]/ul/li[1]/a/span");
   $this->waitForPageToLoad("30000");
   $this->click("//img[@alt='ADD TO CART']");
   $this->waitForPageToLoad("30000");
   $this->click("//div[2]/div[1]/ul/li/a/img");
   $this->waitForPageToLoad("30000");
   $this->type("billing:email", "test@example.com");
   $this->type("billing:firstname", "test");
   $this->type("billing:lastname", "test");
   $this->type("billing:telephone", "8015551212");
   $this->type("billing:street1", "100 Main Street");
   $this->type("billing:city", "Salt Lake City");
   $this->select("billing:region_id", "label=Utah");
   $this->type("billing:postcode", "84116");
   $this->click("//*[@id=\"billing:street1\"]");
   for ($second = 0; ; $second++) {
       if ($second >= 7) break;
       try {
           if (!$this->isElementPresent("//span[@id='billing-please-wait' and contains(@style,'display: none')]")) break;
       } catch (Exception $e) {}
       sleep(1);
   }
 
   for ($second = 0; ; $second++) {
       if ($second >= 7) break;
       try {
           if ($this->isElementPresent("//span[@id='billing-please-wait' and contains(@style,'display: none')]")) break;
       } catch (Exception $e) {}
       sleep(1);
   }
 
   $this->type("merchant_cc_number", "511111111111111");
   $this->click("advice-validate-cc-number-merchant_cc_number");
   $this->type("merchant_cc_number", "4111111111111111");
   $this->select("merchant_expiration", "label=01 - January");
   $this->select("merchant_expiration_yr", "label=2016");
   $this->type("merchant_cc_cid", "432");
   $this->click("//*[@id=\"submit-order-button\"]");
   sleep(1);
   $this->click("//*[@id=\"submit-order-button\"]");
   $this->waitForPageToLoad("30000");
   $this->waitForPageToLoad("");
   try {
       $this->assertTrue($this->isTextPresent("Thank you for your purchase!"));
   } catch (PHPUnit_Framework_AssertionFailedError $e) {
       array_push($this->verificationErrors, $e->toString());
   }
 }
}

Here is the Selenium timeout/maxRunningTime modification code explained.

Testing Disaster Recovery VirtualHosts

me — Tue, 06 Jul 2010 22:05:38 +0000

Maybe you have a bunch of high-volume sites and you want them never to go down? You could get a couple dedicated servers and install Apache and your sites on them. If ever your main server goes down, your plan could be to switch the DNS for your sites over to these DR servers.

How do you test that your sites (set up with VirtualHosts) on these servers work, BEFORE you have to switch them over?

Well, you’d add the new IPs to your local hosts file and test it. But what if you wanted on going tests, maybe with Selenium? Well, you could use a proxy and add the lines to the proxy’s /etc/hosts UNLESS you needed to test the same site on multiple servers. If that was your scenario, you could do the following:

1. install glypeproxy ( http://www.glype.com/ ) which is free for non-commercial use
2. in browse.php, after require ‘includes/init.php’; add:

if (isset($_REQUEST['hostHack']))
{
$goto=trim(rawurlencode(base64_encode($curURL)));
$sessionfixurl="http://YOURURLHERE/hosthack.php?goto=$goto";
header("Location: $sessionfixurl");
}

3. add a new file called hosthack.php:


if (isset($_REQUEST['goto']))
{
$goto=base64_decode(rawurldecode($_REQUEST['goto']));
list($url,$proxyip)=explode("&hostHack=true&hostHackIP=",$goto);
$_REQUEST['hostHack']=true;
$_REQUEST['hostHackIP']=$proxyip;
}
session_name('s');
session_start();
$_SESSION = array();
$_SESSION['hostHack']=$_REQUEST['hostHack'];
$_SESSION['hostHackIP']=$_REQUEST['hostHackIP'];
 
if (isset($_REQUEST['goto']))
{
if ($proxyip!='')
header("Location: $url");
}

4. in includes/settings.php add (after stripObjects):

'hostHack' => array('title' => 'Host Hack',
                        'default' => false,
                        'desc' => 'use a proxy IP for the host name')

5. in themes/default/main.php add this javascript code:

var xmlHttp = null;
 
function updateLocationFirst(pointer)
{
hostHack=document.getElementById('hostHack').checked;
hostHackIP=document.getElementById('hostHackIP').value;
xmlHttp = new XMLHttpRequest();
url="http://YOURURLHERE/hosthack.php?hostHack="+hostHack+"&hostHackIP="+hostHackIP;
xmlHttp.open( "GET", url, true );
xmlHttp.send( null );
 
return updateLocation(pointer);
}

6. in that same file, modify the onsubmit to be:
return updateLocationFirst(this);
7. in that same file, right after foreach ( $toShow as $option ), add:

if ($option['name']=='hostHack') echo("\"hostHackIP\"/>");

8. oh, and in browse, right before curl_setopt_array, add:

$port="80";
if (strpos($this->URL,"https")!== false) $port="443";
if ($_SESSION['hostHack']&&$_SESSION['hostHackIP'])
{
        $this->curlOptions[CURLOPT_PROXY]=$_SESSION['hostHackIP'].":$port";
}

And there you have it. Now the options box will present a hosthack option (what some call the process of overriding DNS by adding a line to /etc/hosts, or in our case, using the proxy method of php’s curl) alongside a box for an IP. If you want to send a hosthacked proxed link, simply postfix &hostHack=true&hostHackIP=THEIP to the end of your glype encoded URL