Cheap PHP Data Structure

One of my favorite things about PHP are the useful built in functions. One set of useful functions is explode and implode.

Explode takes two parameters: a string delimeter and a string to work on. It returns an array of the pieces between the delimeters.

Implode takes two parameters: a string glue and an array. It returns a string of every array element with the string glue in between.

You can do lots of interesting things with these two functions, like create csv files (even though PHP has a function specifically for that), create SQL statements using ‘,’ as glue, or in this case create a cheap data structure. By cheap data structure, I mean data that is easy to create and parse as a structured string. It is definitely not something that should be used on a large scale, but much more suitable for quick experiments.

In this case I needed to send back an unknown number of ids and usernames in a single parameter from Javascript to PHP. I created the string in Javascript like so:

// userArr is an array of arrays of the form (id,username)
var size = userArr.length;
var retStr = "";
for(i=0;i<size;i++){
    retStr += userArr[i][id]+":"+userArr[i][name]+",";
}
// remove trailing comma
retStr = retStr.substring(0,retStr.length-1);
// send retStr
// ...

This creates a string like this:

1:homer,2:marge,3:bart,4:lisa,5:maggie

which is sent as a parameter, in this case the post parameter user_list. When it is received by the PHP page it is parsed like this:

$user_list = $_POST['user_list'];
$user_array = explode(",", $user_list);
$size= count($user_array);
for($i=0;$i<$size;$i++){
    list($id,$name) = explode(":", $user_array[$i]);
    $user_array[$i]['id'] = $id;
    $user_array[$i]['name'] = $name;
}

You can now do whatever you want with this data in PHP. This technique can be very flexible, and with a creative use of delimeters, you could create very complex data structures. But then again, there’s always JSON.

Upgrading to WordPress 3.2

I just completed my upgrade to WordPress 3.2 from 3.1.3. It was a bit more troublesome than it should have been though I’m not sure how much was dependent on my hosting company. I went for the automatic update and it hung for a long time. I tried to wait it out but it didn’t appear to be making any progress so I made the ultimate sin of clicking Upgrade again. As a developer I know that was wrong of me, but I don’t like to wait just like other people!

This ultimately led to receiving a similar error to the following:

Fatal error: Allowed memory size of 262144 bytes exhausted (tried to allocate 77824 bytes)

I got this error on all backend pages making it impossible to make any further changes. The Dashboard would load to a point (with the 3.2 layout) but clicking on any links would result in the above error. I tried the usual attempts at increasing the memory, but none of this, including getting the hosting company involved, helped the issue. I then decided to try a manual update.

This turned out to be the way to go and I will probably do manual updates going forward as I like the control it gives me. I used this guide from WordPress Codex. But to make it very succinct:

  1. Backup all of your files and database
  2. Download the latest version of WordPress that you wish to install
  3. Unzip the download to a location ready for FTP transfer to your server
  4. Upload all the files (not directories) in the wordpress directory into your root wordpress directory, making sure to not overwrite: wp-config.php, .htaccess, robots.txt, sitemap.xml, any other personal files.
  5. Upload and overwrite the old wp-admin and wp-includes directories with your new download
  6. Go to your WordPress admin page, you should be prompted to update your database
  7. Click the Update Database button and you should be taken to your updated WordPress Dashboard

So far I notice a few new features. Most helpful so far is the checkbox to open links in a new window when editing a post. There was a lot of talk on Twitter about the Full Screen editor mode. While it seems to function properly, I really don’t see the need for it? In fact I prefer to have all of my options just one click away and don’t see myself ever using that feature. I do however like the new design and collapsable side navigation.

For my first WordPress update, it was a bit of a headache, though I can’t put the blame on WordPress itself, and once I decided on the manual update, it was resolved in minutes. I would say WordPress succeeded in the update, my hosting and I however need some improvement.

Introduction to Python

I continue to see Python mentioned as a favorite language for many developers. Many high profile companies use Python as one of their main programming languages. Since it is quite popular and held in such high regard, I feel Python is becoming a language necessary to know.

I created a short introduction Python script to learn the very basics and syntax of the language. It can be used from the command line pretty easily. Here is a quick start guide from Python as I will be focused on the language itself and not the setup. Since I use Eclipse for most of my development, I installed the PyDev extension for Eclipse to make my development process easier and everything went smoothly by following their directions. I didn’t need to install an interpreter as Eclipse found the OS default.

Here is my sample code:

'''
Created on Jul 5, 2011
@author: twobee
@summary: a quick introduction to python's basic syntax
'''
# vars and output
stringvar = "Hello World!"
intvar = 3
floatvar = 3.1415926
listvar = ["one","two","three"]
print "Here is a string: ",stringvar
print "Here is an integer: ",intvar
print "Here is a float: ",floatvar
print "Here is a list: ",listvar," with ",len(listvar)," elements"

# if statement
if intvar < 3 :
    print intvar,"is less than 3"
elif intvar == 3 :
    print "It was",intvar
else :
    print intvar,"is greater than 3"

# for statement
count = 0
for elem in listvar :
    print count,":",elem
    count=count+1

# function definition
def print_evens(max=10):
    count = 0
    while(count<=10) :
        print count,    # comma at end prevents newline
        count = count + 2

# call function
print_evens()

Here is the output:

Here is a string:  Hello World!
Here is an integer:  3
Here is a float:  3.1415926
Here is a list:  ['one', 'two', 'three']  with  3  elements
It was 3
0 : one
1 : two
2 : three
0 2 4 6 8 10

Quick Notes
Comments can be rather odd with the triple quote structure for multiline comments and the hash for the single line comments. One interesting feature I found, but didn't use here is that documentation in comments can be accessed within the language itself, similar to a property of the function or object. It is an interpreted language and that lets you get up and running quickly. There appears to be great flexibility in the language and the variables are loosely typed. The lack of typical syntax does make for cleaner looking code. Part of this may be the use of indentation for actual parsing, largely forcing well-formed code blocks (though alternatives are offered as this article explains).

Python Pros

  • Really lightweight feel without seeming incapable
  • Syntax is clearly aimed at streamlining the source code
  • Runs equally well as a web language or general purpose language
  • Appears to have a ton of useful built-in functions
  • Open source and well documented, two often overlooked features

Python Cons

  • No semicolons! It may just be me, but a line of code feels incomplete without a semicolon
  • Using indentation instead of curly braces for blocks of code will have to grow on me
  • No ++ function, very minor and not needed, but I've become accustomed to having it available

Of course all of these opinions are formed after only a few hours of time to work with the basic syntax and none of these points really go too deep into the language. In fact the cons are things that will most likely become second nature after spending some time with Python, but they were my initial reactions. I'll be sure to track my opinion and how it changes with my exposure to Python. In the meantime here were some helpful resources:

Programming Languages: Your Native Tongue

Last year I first started to learn PHP. Within a few months I was in love! I loved the flexibility, readability, surprising power, built-in functions, and of course the documentation. While I’m not an expert, I’ve become quite proficient with PHP in just a year’s time, this is a testament to its usability. But this weekend I was reminded of the importance of being proficient at several languages.

Working on Android, you have to use Java. This shouldn’t be too bad since I used Java through college and a bit beyond. Unfortunately it had been over two years since I had used Java regularly and I was stumbling through the language. I use Eclipse as my IDE and it makes setting up a Java project a few click process. But even the concept of “setting up a project” felt heavy to me. I then had to include external JARs only to compile and get some cryptic error. While the problems were easily resolved, these are issues that you rarely see in PHP (aside from the cryptic errors, I believe that is a programming language standard).

Since I work primarily in the web I tend to value the ability to get something up and running in minutes. Perhaps with enough practice in Java, the same thing could be done, but it would still require more steps and an application server. Understand that I am not bashing Java, it is a fantastic language and ultimately more powerful and scalable than PHP. I was just shocked to find that a language I have used for such a short time, could so quickly replace a language I had used for years. Perhaps PHP was always my native tongue.

Google URL Shortener API

Google provides a URL shortener, which you can find here.

Of course there is an API for this shortener service, so I made a quick example in PHP (note: I’m not a designer ha).

The API can be used without a key but it takes very little time to get one. Just visit the Google API Console, in the drop down Create a new project. Then on the Services tab, turn URL Shortener API to On. Now you can get your key from the API Access tab, labeled API key. To use the key, you pass it as a parameter labeled key.

Here is a sample in PHP:

// create your URL with your key
$url = "https://www.googleapis.com/urlshortener/v1/url?key=[api key]";
// pass in json as a string, note quotes
$data = '{"longUrl":"'.$longUrl.'"}';
// start curl, set url, save output to var
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
// set post method, content type, and data
curl_setopt($ch,CURLOPT_POST,true);
curl_setopt($ch,CURLOPT_HTTPHEADER,array("Content-Type: application/json"));
curl_setopt($ch,CURLOPT_POSTFIELDS,$data);
// run curl, saving response to var result
$result = curl_exec($ch);
// check response code for success
if(curl_getinfo($ch,CURLINFO_HTTP_CODE)!=200){
        // no success, some debugging info is commented out
	//$info = curl_getinfo($ch);
	//print_r($info);
	//print_r($result);
	return false;
}
curl_close($ch);
// turn returned json into an object
$json_obj = json_decode($result);
//print_r($json_obj);
// print the field that stores the shortened url, id
echo $json_obj->id;

I included what might be some helpful debugging code, just commented out. Curl in PHP is very resourceful and improvements are appreciated.

Google+: Cautious Optimism

Yesterday was the big announcement of Google+, Google’s latest attempt at social. As a fan of Google I am really excited about this product, then I remember the past. You may remember the recent failures of Wave and Buzz, with Buzz even starting some legal issues. I am hoping that they learned a lot from those past failures as Gina Trapani pointed out on smarterware.org.

Currently Google+ is in a relatively closed field test and this is the first good news in my opinion. With the past failures in social and the amount of resources seemingly thrown at social, it is more important for Google to succeed here than in the past. It appears that last night they opened up Invites to more people and this should increase the feedback that we get about Google+.

Since I’m still waiting for an invite, I can’t go into any details beyond what I have heard or seen from others. Some great information about Google+ can be found in This Week in Google, episode 101: Inside Google+. Based on the tour and reading others first impressions, I made a few observations.

First, the concept of Circles is something that I believe most people want in a social network. The problem is that established social networks face the uphill battle of getting their users to go through their list of existing friends and categorize them. This is an advantage for Google+ as your list is empty and Circles will be a part of the process of adding friends, rather than adding friends, then categorizing them.

Hangouts are easily my favorite feature. I have considered a set up similar to this in the past and even recorded some details of what I thought might be obstacles as well as what features may be most useful. Hangouts are essentially group video chats, but to limit it to that would be foolish. Some ideas to consider are: shared movie viewing (apparently they have this integrated already with YouTube), shared web browsing, screen sharing, group document editing with video, and all the other things you wished you could do while doing a simple text chat with someone. I believe Hangouts have the highest risk/reward as I think it could change the way social is done on the web or it could become your run of the mill video chat.

There are other features such as Stream, your typical social feed, Huddle, group texting and currently mobile only, and Sparks, interest based recommendations and hopefully conversation starters, as well as a few other details. A great resource I found was the support page.

If anyone has been using Google+ and has some opinions to share let me know. Also if you have an invite, you know what to do with it!

PHP Variable Names with a Dollar Sign

I was recently working on a project to get a list of contacts from a users Gmail account. After pushing through the OAuth process I was able to get a JSON response with the information I needed. Using json_decode, I created an object from the JSON, unfortunately, the object contained objects with names like; “gd$email”.

Here is a simplified version of the object:

{
     ["version"]=> "1.0"
     ["encoding"]=> "UTF-8"
     ["feed"]=> {
          ... some of the users data ...
          ["entry"] => {
               [0]=> {
                      ... some contacts info ...
                      ["gd$email"]=> {
                           [0]=> {
                                ["rel"]=> "http://schemas.google.com/g/2005#other"
                                ["address"]=> "my@email.com"
                                ["primary"]=> "true"
                           }
                      }
               }
               ... more contacts similar to first entry ...
          }
     }
}

To access the email address I attempted to use an object call like this and got the following error:

$email = $json_obj->feed->entry[0]->gd$email;

Parse error: syntax error, unexpected T_VARIABLE

A quick google search did the trick and I found the curly bracket syntax for variables. I was unable to find any in-depth documentation on this syntax, but this is a good overview from PHP documentation on variable variables. In general it appears that whatever is in curly brackets is evaluated as a string and used as a variable name. This is really great flexibility in PHP and something I’ve needed in the past, but it is something I feel should only be used when there is no other solution. My final working code looked like this:

$email = $json_obj->feed->entry[0]->{'gd$email'};

my@email.com

This was a great find for me and something I’d like to experiment with in the future. It should be noted that you can put (what appears to be) any statement you want into the curly braces, including logic.

Additional Resources:

WebSockets: phpwebsocket issue

Part of the HTML5 spec is the WebSocket API. It allows bidirectional communication with a server. The main advantages of websockets is the reduction in latency and protocol overhead. For an in-depth comparison of websockets and past techniques like long polling, check out the following article: HTML5 Web Sockets: A Quantum Leap in Scalability for the Web.

Although websockets are still in development, both the protocol and the API not yet finalized, I think this is a technology that will get enough support from developers to eventually stabilize. So I have begun experimenting with websockets using the phpwebsocket as my server.

Recently, I ran into an issue where the first message a client received was not being handled correctly. There did not appear to be any errors, the client just acted as if it did not receive a message, though I was sure it got sent server side. Luckily I stumbled on this post on StackOverflow describing my exact problem and its solution.

The Problem

Receiving and processing the first message that a client is sent

The Solution

Change line 102 (if unedited) in websocket.class.php:

From:  socket_write($user->socket,$upgrade.chr(0),strlen($upgrade.chr(0)));
To:      socket_write($user->socket,$upgrade.chr(0).chr(255),strlen($upgrade)+2);

For some details on the issue you can check the issue tracker for phpwebsockets (issue ID 16), but essentially the websocket handshake wasn’t considered complete because it hadn’t received the 0×00 and 0xFF (chr(0) and chr(255)) bytes to end the handshake though I don’t see any mention of that in the protocol document at first glance. Anyone know why adding those two bytes at the end works?

US Government Spies on Social Networks

Just ran across this article posted by @ioerror: Project PM Leaks Dirt on Romas/COIN Classified Intelligence Mass Surveillance. I was immediately appalled and read as much as I could in a short time (links at the bottom). The operation was revealed by Barrett Brown under the Project PM operation. Check back here for what will apparently be a detailed description of his findings in the near future.

Project PM found that HBGary Federal along with several other government contractors were found to be part of program Romas/COIN, a large-scale social network monitoring “tool” that uses relationships and natural language analysis, among other things to create identifiable profiles of individuals. In addition to this, the Romas/COIN program is about to get an upgrade, known as Odyssey. The breadth of the surveillance seems very large and includes companies such as Akami, Apple, AT&T, and even my beloved (apparently no longer) Google. For all the gritty details be sure to read the above article, it is important to your personal health, trust me.

How was this all found? Why through the works of Anonymous of course! On February 6th, 2011 Anonymous released the emails that they had acquired from HBGary. Using these emails Project PM was able to tie together certain details and draw the above conclusions, as well as many others. Most notably, how these federal contractors were willing to take money from corporations to take out their enemies through the use of “clandestine and dishonest means”. To put it bluntly, you are paying taxes for the government to spy on you. This is why I support groups like Anonymous and LulzSec to an extent. They can cause collateral damage (particularly LulzSec), but it seems to be effective in some ways. In general I support “hacktivism” as a means to promote change and growth, it just needs to be done more responsibly than it is done now.

Our digital rights and privacy are disappearing at an alarming rate. In addition to this recent news, is this bit about FBI raiding a data center in Virginia and arbitrarily grabbing three server racks. They knew that only one was of interest but rather than take the time to investigate they decided to act rashly and take down several innocent, unknowing sites in the meantime, not to mention the stress and work that must have fallen on those Sys Admins for DigitalOne. This is a sign of ignorance, total disregard for others, or both.

You may read these cases and think that you have nothing to hide and therefore nothing to worry about. WRONG! This is the common problem with rights and privacy as a whole. Always remember that a legal action today can quickly become an illegal action tomorrow, and it is all up to the government at the time, not us.

Other links:

Bash Backup Script

One of the main reasons I learned shell scripting is for automated backups. During my research I found several useful examples that are linked below, but none of them addressed my exact needs. I needed to tar the files and mysqldump the database, in one script, and the MySQL DB is on a different host. I ended up using the below script. (Uninterupted script below)

A few things to note, the scripts must be saved with the .sh extension, they must have execute priveleges (0755 on the file or directory), the destination directories need to have write access, and finally the destination directory for your backups should be above your web root so that it can’t be reached through the web (you can put on additional restrictions if needed).

#!/bin/bash

# script to backup the web directory
# and the MySQL database

Set up the script and make comments

#set up some variables here
date=$(date +%Y%m%d%H%M)
# not sure why this needs quotes but it does?
source="/home/source/directory"
dest=/home/backups/bk_$date.tgz

Set up the source and destination file directories

#database settings
db_host=mysql.host.com
db_user=db_user
db_pass=db_pass
db_name=db_name
sql=/home/backups/db_$date.sql

Set up database variables

echo "Starting backup of $source ..."

#need to research the options for tar
tar -czvf $dest $source

Backup files to a tar ball using the tar command

# $? contains the return message of the last run program (tar in this case)
if [ $? == 0 ]; then
   echo "Finished backup of $source to $dest"
   echo "SUCCESS"
else
   echo "Unable to back up $source"
   echo "FAIL"
fi

Check the results of the tar command and display user friendly message

# start the database backup
echo "Starting backup of $db_name ..."

mysqldump -c -h $db_host --user $db_user --password=$db_pass $db_name > $sql

Make a database dump to a sql file

if [ $? == 0 ]; then
   echo "Finished backup of $db_name to $sql"
   echo "SUCCESS"
else
   echo "Unable to backup $db_name"
   echo "FAIL"
fi

echo "Script complete"

Check the results of the mysqldump command and display user friendly message

There is tons of room for improvement, including the ability to delete old archives in a “smart” way, compare previous versions and only back up differences, create crontab so that automation of the script is independent of the host (currenly I have this configured to run automatically through my hosting control panel), and other fallback options.

Here were two of the links that helped me through this script:

#!/bin/bash

# script to backup the web directory
# and the MySQL database

#set up some variables here
date=$(date +%Y%m%d%H%M)
# not sure why this needs quotes but it does?
source="/home/source/directory"
dest=/home/backups/bk_$date.tgz

#database settings
db_host=mysql.host.com
db_user=db_user
db_pass=db_pass
db_name=db_name
sql=/home/backups/db_$date.sql

echo "Starting backup of $source ..."

#need to research the options for tar
tar -czvf $dest $source

# $? contains the return message of the last run program (tar in this case)
if [ $? == 0 ]; then
   echo "Finished backup of $source to $dest"
   echo "SUCCESS"
else
   echo "Unable to back up $source"
   echo "FAIL"
fi

# start the database backup
echo "Starting backup of $db_name ..."

mysqldump -c -h $db_host --user $db_user --password=$db_pass $db_name > $sql

if [ $? == 0 ]; then
   echo "Finished backup of $db_name to $sql"
   echo "SUCCESS"
else
   echo "Unable to backup $db_name"
   echo "FAIL"
fi

echo "Script complete"