ORM or Not? Part Two - Definitely Not

Maybe that's a little too harsh - but I don't think so.

Here's the background:

One of the biggest recurring programming problems I've had to deal with in site and application development has been - (drum roll) - developing the database. Not populating it - that's just boring. Developing and refining the data definitions.

The "Conventional Wisdom" promulgated by Software Gurus is basically this:

1. Databases are Good With Data
2. Data Objects Are Good With Data
3. A Programmer Must Define a Mapping Between these Two Good Things and then All Will Be Good

Not my experience.

Software Gurus don't write application. Software Gurus write Books and (occasionally) toy examples. So they really don't understand that Good does not generalize because the Good that Databases are with data is all about safety and accessing it in great and small piles. The Good that Data Objects are about is manipulation, fine structure, and flexibility.

My experience with Object Relational Data Mapping - which is just a fancy phrase for how you get the data from the database into an object and back - is this: when I Did it Their Way, I had to hand-coordinate both my Objects and my Database.

So when I wanted to add a field or change a field or change a datatype, I had to do everything twice.

Now Rails has Active Record - which is a stupid Software Guru name for an ORM which creates the Data Objects automatically (my drivers license is an active record - meaning it's current so I can legally drive), but that doesn't 'solve the problem' it just moves it. [maybe Rails 3 is smarter,
I dropped out just as Rails 2 was coming out]

Specifically, the Rails solution moved the problem to 'migrations' which turned out to be fragile. I know: I tried it.

So, a couple of years ago I proposed Not building an ORM, but rather loosening the coupling between the Database and the Data Objects.

Here's what I've done:

1. Defined an PHP class which implements a data abstraction which includes most of the types of data and methods I want in my CMS [see http://www.yasitekit.org] and knows how to map those data types into database fields. It also knows how to create the database structures, create, update, delete, and retrieve those values.
2. Defined a PHP class which makes it easy (relatively speaking) to create objects which are instances of the data abstraction class. These classes come with a bunch of methods which I've found useful as well as a management class which automatically provides interactive data administration.
3. Some automated analysis tools which work with this stuff so it's possible - and relatively painless and safe - to add, delete, and modify the derived objects (point 2) and reload the database. All done by hacking the derived PHP objects, but never ever touching SQL.
4. As a side benefit, I threw together a database adaptor which abstracts the 8 essential database operations [create/destroy database;create/drop table;insert,update,delete,select] at a level high enough that we never have to muck with SQL. This makes it possible to augment it with non-SQL databases - such as Mongodb. In it's present form it allows painless migration between sqlite, mysql, and postgresql (it currently handles 5 different PHP database interfaces - automatically figuring out what's available - yada yada yada)

So, at this point, I feel more than comfortable saying No to ORM's.

BTW - if you go to the YASiteKit site, you may be a little dissapointed because all of this good stuff is not available yet. But It Will Be - Real Soon Now (no kidding). And a whole lot more.

What is on the site is a pretty-much full set of documentation - both about the system design and about how each of the parts work [I write doc as I go, so it's not all pretty - but it's useful (I know because I use it)].

So take a look. If you have any comments - let me know.

Also, if you like it enough to want to help - let me know sooner - I'm almost ready to turn it loose to see if it gets any traction.

Timing Tests

OK - I'm not a timing test expert. I'm not a program profiling expert. etc etc etc

I know timing tests are 'hard to do right'. I know that there are all kinds of consideration. I know that 'to do it right, you have to . . .'

But I don't really care about 'doing it "right"' according to some picky standard.

What I do care about is not writing really slow code.

For that, the rules are simple:

Rule 1: Don't do things that take a long time

Rule 2: If you have to do repeat something a lot, check out alternative ways to do it and pick the method which is both clear to read / understand and takes the least time.

Rule 1 - expanded:

You do this by knowing how long things take and which operations are blocking. You don't need to be accurate, because for most things, 'takes a long time' is measured in orders of magnitude.

Here are the relevant cases:

1. Monolithic program doing in-memory data access/processing
   
2. Multi-threading/parallel processing/whatever - any form of parallel processing which is executed inside a single process context. Here you tend to lose because of blocking and communication - one thread needs to access shared data and so blocks reads, etc OR one thread needs results from another to continue OR etc.
3. Self generating code - aka Metaprogramming. This is a cool way to impress your friends, but it costs orders of magnitude in performance. The idea is to write code which traps function calls to functions which don't exist, then parse the function name and build a function 'on the fly' to do the task encoded in the function name, dynamically build the call sequence, execute the function and return the result. It's not hard to do, but it's pretty much unnecessary (almost all the time) and really slows things down, because parsing strings takes a lot of repetitive work. That's why we have compilers!
   
4. Disk reads are much longer - at a minimum they require a context switch as you make a system call. Then it depends on file size, caching in the operating system, memory size, etc. The Rule is: for repeated reads, try to do only Once and cache the result in a variable
5. Run a subprocess - this requires a context switch, process invocation, lots of disk reads, etc etc followed by receiving result, parsing it, etc etc. Much more expensive than disk, but less expensive than Network reads.
   
6. Network reads are longest - not only do they require a system call, you typically have to run another process someplace. If that process is on a different host, then the cost is astronomical relative to in-memory and disk i/o.

So Rule 1, says - if you don't really need to do the Slow Thing, then don't. And Do the Slow Thing as seldom as you can get away with.

Rule 2 - expanded

Right now I'm writing a lot of PHP (don't groan, it seemed like a good idea at the time) and in this code I have lots of places where I need to do things based on the value of a string. For example, I'm writing a lot of PHP5 objects where I put guards on attribute access so that I can find spelling errors (my High School English teachers understand why I need to do this).

So I have lots of functions that look like:

function __get($name) {
if (in_array($name, array('foo', 'bar', 'baz'))) {
  return $this->$name;
}else {
  throw new Exception("$name is not a valid attribute name");
}
}

or

function __get($name) {
if (($name == 'foo' || $name == 'bar' || $name == 'baz'))) {
  return $this->$name;
}else {
  throw new Exception("$name is not a valid attribute name");
}
}

or

function __get($name) {
switch ($name) {
  case 'foo':
  case 'bar':
  case  'baz':
    return $this->$name;
  default:
    throw new Exception("$name is not a valid attribute name");
}
}

I do this a lot, so I need to know which one is the fastest. I don't need to know precisely, I just need to know 'more or less'.

To do this, I need to build a test case and run it to get some timing numbers.

The test case doesn't have to be perfect, but it does need to put the emphasis on the differences between the three different methods. It also has to be large enough to be able to distinguish run times between the methods.

In this case, I built the three functions each with about 150 alternatives and the built a list of trials which would fail about 1/2 the time. I then executed each function a bunch of times.

How many is the right bunch? I'm lazy, so I start small for the number of repetitions and then crank it up until the total run time per method is around 10 to 60 seconds.

Here's what I got:

• switch: 49.7731 seconds
  
• in_array method: 86.3004 seconds
  
• if with complex conditional: 57.0134 seconds

Guess which method I'm going with.

[guess how I'm going to refactor a lot of my code (sigh - I should have tested first)

News Flash: PHP Documenters Insane!!!!

The PHP documentation has gone from very useful to hideously obstructive.

The people who are rearranging the doc into little, tiny chunks which are hyperlinked all over the place obviously never write code.

I just spent 10 minutes trying to find the name of an IO Exception so I can use it in some code I'm writing.

Old Doc:

1. I would go to the index, click on Exceptions and then scroll down the page (or do a find on IO) and there it would be. 10 seconds tops.

New Doc:

1. Go to the index click on Predefined Exceptions
2. Click on Exception - find description of Exception Object - info not there
3. Back Button
4. Click on Error Exception - find description of Generic ErrorExeption object
5. Back Button
6. Click on SPL Exceptions (what the hell is this? - something new?)
7. Look at Table of contents: 13 Exception Categories - none of which
8. looks like an IOException
9. Click on Predefined Exceptions in the See Also -
10. Back to Previous Useless Page - And Repeat
    

First they completely screw up the Perl Regular Expression page by chopping it into tiny, obscure chunks and now you destroy the exception documentation.

To the PHP Documentation Project:

PLEASE put it back the way it was.

Or get somebody who actually uses this stuff like a handbook while writing code to fix it

Or shoot somebody.

To Everybody Else:

Maybe the documentation people have stock in a book company and want the reference books to succeed by making the online Doc unusable?

All I can say is that the way they are going is really going to help Rails and Django.

What do you think?

P.S. Please Send a Nasty Note to the Gods of PHP

PHP, Language Design, and Confusion

References

When I wrote some iterator code which returns a reference to an internal array in an object I discovered an interestingly easy to introduce bug. Without getting into the details, I left out a '&' and clobbered my underlying data structure.

Here's the offending loop:

for ($row=&$obj->firstRow();$row;$row=$obj->nextRow() {
  other stuff
}

where firstRow() and nextRow() are iterator methods defined on the object which do exactly what they sound like. [oh yeah, they both are defined as returning references]

Spot the bug?

I left off the '&' between the '=' and the '$' in the update part of the for loop. This causes $obj->nextRow() to overwrite the firstRow.

Strangely enough, if I wrote this in C I wouldn't have made the error. The equivalent C code would be:

for (ptr=*obj->firstRow();ptr;*ptr=*obj->nextRow() {
  *ptr-> whatever
}

Notice that I have to explicitly dereference the pointer ptr in order to clobber the initial element.

So this covers PHP and the Confusion.

Here's where the Language Design comes in:

C makes an syntactic distinction between accessing a pointer and dereferencing it.

PHP doesn't. If a variable contains a reference then assignment deferences it implicitly and silently.

Consequently, you can't really understand a chunk of PHP without reading all the definitions of the variables which preceed the chunk you're looking at. This is a bad thing and it violates that tried, true and mostly forgotten maxim of 'code locality'.

This also leads to a lot of bugs - and 'Bogus' bug reports - involving references. [just check out bugs.php.net - 655 Bogus bugs relating to 'reference']

I think PHP would be a lot easier to understand if there was a syntactic difference between assigning to a variable and assigning to the referrant of a variable.

Practically speaking, I don't think it will happen because of 'backward compatibility' and 'NIH'.

What I'd like to see is a dereference prefix which is accepted in PHP 5.x and becomes mandatory in PHP 6. There should also be a warning option which generates a warning if a variable containing a reference is assigned value and the variable is not prefixed.

Here's my first (and only) choice:

&$foo

This is currently used to denote a reference when used on the right side of an assignment. It should be a simple matter to extend this to dereferencing on the left side and in expressions.

Efficient Code - PHP

I guess I'm a nut about execution speed. Well, not if it gets in the way of clarity, but I hate writing slow code.

But . . . always a 'But' . . . it's hard to know what is slow and what isn't. Seems like it should be easy, but I never know until I measure.

Here's an example:

The PHP manual page for preg_split() says that you shouldn't use it unless you need the flexibility of regular expressions. You should use explode() or str_split(), because they are simpler and therefore faster. I believed that.

Then I thought I'd like to see how much faster, so I wrote a couple of loops which split a comma separated string with both preg_split() and explode() to see how much I was really losing.

Here's are the average times of 10 passes through 1,000,000 splits using each:
preg_split() - 3.51239209175 seconds
explode() - 4.33661820889 seconds

preg_split() is about 23% Faster than explode().

I'm still not sure I believe it. Here's the code - so you can try it yourself.

And, please let me know if you see anything I did wrong.

$str = 'this, is , a , string, with , commas, in ,it';
$count = 1000000;
$passes = 10;
$times = array(xdebug_time_index());
$avg = array('preg' => array(), 'explode' => array(), 'map-explode' => array());
function dt($offset = 1)
{
  global $times;
  $last = count($times) - 1;
  return $times[$last] - $times[$last - $offset];
} // end of dt()

for ($pass=0;$pass<$passes;$pass++) {
  echo "Pass $pass\n";
  $times[] = xdebug_time_index();
  for ($i=0;$i<$count;$i++) {
    $v = preg_split('/^\s*,\s*/', $str);
    unset($v);
  }
  $times[] = xdebug_time_index();
  $avg['preg'][] = dt();
  echo "$count preg_split's(regx, str): " . dt() . "\n";
  
  $times[] = xdebug_time_index();
  for ($i=0;$i<$count;$i++) {
    $v = array_map('trim', explode(',', $str));
    unset($v);
  }
  $times[] = xdebug_time_index();
  $avg['map-explode'][] = dt();
  echo "$count array_map(trim, explode(',', str)): " . dt() ."\n";

  $times[] = xdebug_time_index();
  for ($i=0;$i<$count;$i++) {
    $v = explode(',', $str);
    unset($v);
  }
  $times[] = xdebug_time_index();
  $avg['explode'][] = dt();
  echo "$count explode(',', str): " . dt() ."\n";
}

echo "Averges\n";
foreach ($avg as $key => $ar) {
  echo "$key average of $passes trials of $count splits: " . array_reduce($ar, create_function('$a,$b', 'return $a+$b;'), 0) / count($ar) . "\n";
}

Singleton Patterns in PHP 5

The Singleton Pattern is one of the few things I really like from Design Patterns. While the original Gang of Four book has some very good stuff in it, later works seem like Software Industry Marketing Fluff [SIMF] - but I've already bitched about that.

The Singleton is GREAT because it can replace Global Variables - if not everywhere, at least in a lot of important cases.

In case you don't know what a Singleton is, it's an Object which has only one instance. Ideally, you would like to write something like

$foo = new Thing();

and get a reference to the one instance of Thing.

The "normal" way to implement the Singleton Pattern is to hack up the class definition in an unnatural way:

• Disable the normal constructor by making it private
• Define a class variable to hold the One Instance
• Create a different public function which is used to instantiate the first instance and return it on all subsequent 'instantiations'

This is because none of the Object Oriented Languages directly support Singleton - except Ruby which has a Singleton base class so you can create them by normal inheritance.

I was so taken with Ruby's Singleton base class, I decided to write one in PHP 5.

Drum Roll........

Didn't work. Failed Miserably.

It turns out that in PHP 5.2 static class variables [that is, Class Variables] are always in the base class. I won't go into that here. Instead, take a look at this test I wrote.

Anyway, just because we can't build a base class for Singleton Objects, doesn't mean we can't clean up their creation. Here is an example of a pseudo-Singleton which allows me to write

$foo = new Thing();

and have a $foo object which acts just like a Singleton even though it has many instances.

The reason this works is that we don't need a single instance of the Object. What we need is a reference which returns the common data, where the methods act on the common data, where modification of the common data is seen by all existing references.

The Pseudo-Singleton looks, acts, and smells like a Singleton, except it is more convenient and natural to use.

The only thing better would be a 'singleton' keyword prefix to use in 'class' definitions.