Expressed Opinions

The Blog has Moved!!!

2011-05-18T18:08:00.003-06:00

I've moved this blog to http://mikehoward-musing.posterous.com

Thanks for the fish

Mike

Creating a Rails 3.0 Gem

2011-04-26T11:28:00.001-06:00

Building a Gem for Rails 3

This is a 'work in progress' - any and all corrections, comments, suggestions
welcome!

Believe it or not, it’s not that hard.

Here’s the basic outline:

lay out a basic Ruby gem, with the normal directory structure
decide how you need to hook into Rails
- if you need to patch some of Rails internal structures - such as
  add some functionality to ActiveController - then you need to write
  a Railstie
- if you need to add a controller, a model, view, rake task or a generator -
  then you probably should use an Engine. [the difference between a Railtie
  and an Engine is that an Engine is a Railtie with more stuff]
- Or, if you want to embed an entire Application into another, you can use
  subclass Rails::Application. If you need to write an Application, then
  you need to find somebody who knows how to do it.
- Or, if you want to make a Plugin, you should forget it and just build a gem
  and either implement a Railtie or an Engine
build, test as usual. Below I’ll show how you can hook your local, development
gem into a locally run Rails app.
package and ship out to github.com and rubygems.org

In everything which follows, I’m assuming that you are adding functionality to
Rails which requires some sort of ‘initialization’.

What does ‘requires initialization’ mean?

Ok - when does your gem ‘require initialization’?

if you need to run a rake task or a generator to install some stuff in order for
your gem to work - then it ‘requires initialization’. In fact, you should probably
create an Engine.
if you’re adding controllers, views, etc, then you need an Engine
if you want to monkey patch ActiveController or one of the other basic Rails
classes, then you will want to ‘include MyGemModule’ into that Rails class when
it is autoloaded. For that you need to insert yourself into the autoload sequence
and so you ‘require initialization’. Specifically, you need at least a Railtie.
finally, if you’re not doing any of this and there is some way for your gem
to provide services without any startup initialization and without living in any
namespace Rails knows about, then you don’t need initialization.

So, the answer is - pretty much all gems which will extend Rails are going to
‘require initialization’ and your initialization stuff goes into your Railtie
or Engine.

So …

What goes into your Railtie / Engine

Surprisingly little - but figuring out what that ‘little’ is can be
daunting.

Here’s the Railtie I wrote for my manage_meta gem:

module ManageMeta
  class Railtie < Rails::Railtie
    initializer "application_controller.initialize_manage_meta" do
      ActiveSupport.on_load(:action_controller) do
        include ManageMeta
      end
    end
  end
end

There’s a lot going on here in 9 lines (typical Ruby compactness!)

Before going into this code, it will help to put it in the context of how
this gem is structured.

the gem and github repository are both named manage_meta
all the code which does something is in manage_meta/lib
the file which is required is named manage_meta/lib/manage_meta.rb. All it does
is require files from manage_meta/lib/manage_meta/.
- It always requires manage_meta/lib/manage_meta/manage_meta.rb: this allows
  me to write unit tests which are independent of Rails
- It conditionally requires the Railtie.

Here it is:

require 'manage_meta/manage_meta'
require 'manage_meta/railtie' if defined? Rails

OK - so all we need to do is get our Rails app to include manage_meta/lib/manage_meta.rb.
But this is a distraction right now. We’ll get back to it when we go
over the Rails boot process.

By the way, the names are significant!

The root directory manage_meta, the top-level require file manage_meta/lib/manage_meta.rb,
and the subdirectory of lib manage_meta/lib/manage_meta/ need to all be
the gem name.
I think that the Railtie (or Engine) really needs to be named railtie.rb
(or engine.rb) in order for Rails to find it. [this is a guess which I may
never resolve because: 1. it works when I do it like this and 2. there’s no reason
not to. If anybody can confirm this, I’ll remove this caveat and give them credit]
module ManageMeta - I’m extending my module with some Rails specific code.
While not visible here, this is conditionally included in lib/manage_meta.rb

OK, back to the Railtie:

module ManageMeta

Your gem must be namespaced to a Module and you have to define your Railtie (or Engine)
within that module.
class Railtie < Rails::Railtie (or class Engine < Rails::Engine)

Sure, you can call it Bob if you want, but why bother? It’s real name is
ManageMeta::Railtie or MyGem::Railtie - which is safely namespaced, so there
won’t be any conflict here.

The important thing is that this get’s you all the stuff in Railtie - most importantly,
for what we’re talking about here, this is where you get the ability to initialize
by calling initializer [defined in rails/lib/rails/inializable.rb (see the pattern?
everything is a gem)]
initializer “applicationcontroller.initializemanage_meta” do …

initializer takes a name, a block, and (optionally) a couple of options. The
option keys are :before and :after and we’re going to ignore them for now.

It builds an Initializer instance and stuffs it into the array-like object named
initializers. I say it’s array-like because all the Initializer instances are
in sequence, but the options allow anyone in the know to place their specific
Initializer.

Let’s just take it on faith that as Rails boots, it goes through initializers
and runs the code blocks we pass in.

Here’s the initializer code:

def initializer(name, opts = {}, &blk)
  raise ArgumentError, "A block must be passed when defining an initializer" unless blk
  opts[:after] ||= initializers.last.name unless initializers.empty? || initializers.find { |i| i.name == opts[:before] }
  initializers << Initializer.new(name, nil, opts, &blk)
end

ActiveSuport.onload(:actioncontroller) do ; include ManageMeta ; end

Finally we get to the place we were going.

ActiveSupport seems to be pretty much a support library which just lays around like
a sleeping dog until you poke it to do something. Then it does it and goes back to
sleep.

One of the things ActiveSupport provides is support for creating queues of code
blocks which can be ‘run later’. This support is defined in
active_support/lib/active_support/lazy_load_hooks.rb - which defines 3 class
methods:
- on_load(name, options, &block) - which is we use to add our code to a hook
- runloadhooks(name, base = Object) - which runs all the defined hooks. We don’t
  touch this - but it’s really important.
- execute_hook(base, options, block) - which we never touch - it’s used by
  run_load_hooks to actually run the code block on the object

So, here’s the short version of what those 9 lines do:

It adds the statement include ManageMeta to the autoload sequence which is triggered
when ActionController is first loaded.

It does it by adding that code to the on_load chain keyed to the name :action_controller.

Let’s Try to Generalize This

OK - so how did I know how to do this?

Code crawling - of course.

I grep’ed the Rails 3.0.6 gems for calls to run_load_hooks and found the following
keys defined:

:action_mailer
:action_controller
:action_view
:active_record

:i18n
:before_initialize
:before_eager_load
:after_initialize
:before_configuration

The first four (action_mailer, action_controller, action_view, and active_record)
all call run_load_hooks at the end of ::Base. So if you want or need
to hack these base classes, you just copy the code above and use the appropriate
name.

So far I haven’t found a need to hook into any of the other 5, so I’ll just leave
those as an exercise for the reader. [send me a link to what you find and I’ll
add it here]

But but but … What about Engines?

Well, one place you need an Engine if you have some stuff in the app directory.

So what you do is add an app directory to your gem, structure it like a Rails
app directory - and it “just works”.

Another place is if you want some rake tasks.

What you do then is create a directory called manage_meta/lib/tasks/ and put
your rake tasks in files called a_task.rake - and it “just works” too.

Another place is when you need a generator.

Here you need to call the generate

I haven’t messed with building a generator yet - but it’s on the list.

As soon as I get to it (and have at least a less vague idea of what to do), I’ll
add some stuff here

But … But … But … How does Rails Find my Gem and All this Goodness?

Glad you asked.

The answer is “The bundler did it!”

If you look in your Rails app in config/boot.rb, you will find a file which
contains something like:

require 'rubygems'

# Set up gems listed in the Gemfile.
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)

require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])

Here’s an easy way to see what this does.

This is my Ruby environment in the top directory of a Rails site I’m working
on. [I’m using rvm, so it shows the ruby and stuff I have set up for this site]
This is just simply running irb, so the bundler setup stuff isn’t run.

Here’s the default $LOAD_PATH

mike:clovetech2 mike$ irb
ruby-1.9.2-p180 :001 > puts $:.join "\n"
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/x86_64-darwin10.7.3
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin10.7.3
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/vendor_ruby
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1/x86_64-darwin10.7.3
 => nil

Here’s what $LOAD_PATH looks like after bundler/setup is required. All those
extra files are pulled in from the Gemfile.

Most of the new stuff are gems which I’ve pulled from rubygems.org and which
are stuffed rvm’s gemset for the Ruby I’m using. It also adds in my development
gems. More below

mike:clovetech2 mike$ rails c
Loading development environment (Rails 3.0.6)
ruby-1.9.2-p180 :001 > puts $:.join "\n"
/Users/mike/Rails/clovetech2/lib
/Users/mike/Rails/clovetech2/vendor
/Users/mike/Rails/clovetech2/app/controllers
/Users/mike/Rails/clovetech2/app/helpers
/Users/mike/Rails/clovetech2/app/mailers
/Users/mike/Rails/clovetech2/app/models
/Users/mike/Rails/Mikes-Gems/use_tinymce/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/webrat-0.7.3/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/sqlite3-ruby-1.3.3/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/sqlite3-1.3.3/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/rails-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/railties-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/thor-0.14.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/nokogiri-1.4.4/lib
/Users/mike/Rails/Mikes-Gems/manage_meta/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/database_cleaner-0.6.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/autotest-rails-4.1.0/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/annotate-models-1.0.4/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/activeresource-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/activerecord-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/arel-2.0.9/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/actionmailer-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/mail-2.2.15/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/treetop-1.4.9/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/polyglot-0.3.1/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/mime-types-1.16/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/actionpack-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/tzinfo-0.3.26/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/rack-test-0.5.7/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/rack-mount-0.6.14/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/rack-1.2.2/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/erubis-2.6.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/activemodel-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/i18n-0.5.0/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/builder-2.1.2/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/activesupport-3.0.6/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/abstract-1.0.0/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/ZenTest-4.5.0/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@global/gems/rake-0.8.7/lib
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/bundler-1.0.10/lib
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/x86_64-darwin10.7.3
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin10.7.3
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/vendor_ruby
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1/x86_64-darwin10.7.3
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/actionpack-3.0.6/lib/action_controller/vendor/html-scanner
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/rack-mount-0.6.14/lib/rack/mount/vendor/multimap
/Users/mike/.rvm/gems/ruby-1.9.2-p180@clovetech2/gems/rack-mount-0.6.14/lib/rack/mount/vendor/regin
 => nil

Take a look at /Users/mike/Rails/Mikes-Gems/use_tinymce/lib
and /Users/mike/Rails/Mikes-Gems/manage_meta/lib. Those are my development
gems.

bundler has hooked my development gems into the load path.
I pulled that off by specifying those gems using:

`gem "manage_meta", :path => "~/Rails/Mikes-Gems/manage_meta"`

Take a look at the bundler site’s
for a discussion of the Gemfile
options.

And Wait … There’s More

Notice that only the lib directories for each included gem are added to
the $LOAD_PATH.

So how does the Engine get to the app directory and how does rake find
rake tasks?

The answer - as usual - is convention.

Directory structure is important. app has to be locatable in a standard way
using File.expand_path. For example, I strongly suspect that Rails::Engine
contains something which effectively does:

$LOAD_PATH << File.expand_path('../../../app', __FILE__)

(it doesn’t look like this - Rails is far more generic and self-programming than
something this straightforward)

Anyway - the thing to realize is that - for mere mortals - it’s best to stick
to the “correct” directory structure and naming conventions.

So if you build it and it doesn’t work - check your names and directory structure.

A Little about Initializers

railties/lib/rails/initializable.rb defines two classes, some class methods,
and an instance method:

The classes are:

Initializer - which is a named container for a block of code. It has provision
for a context [which turns out to be a binding in which to run the code block]
and before and after properties which are used to provide a sorting order.
(more below)
Collection < Array - which is an array which can be sorted using the tsort
method in the standard Ruby library. Sorting uses the before and after
properties.

The class methods are:

initializers - which is a Collection
initializers_chain - which constructs an initializers instance for the class
which calls it. Specifically, it consists of all ancestor classes and modules
which respond_to? initializer [in reverse order, so the the initializations
are processed from the top of the class/module hierarchy down to the lowest
descendent (which is self)]
initializers_for(binding) - returns an initializers chain where each
Initializer instance is bound to binding
initializer(name, opts = {}, &blk) - whom we’ve seen before up here

The instance method is:

initializers - constructs and caches the initializer_chain constructed for this
instance of the class which mixes in Rails::Initializable.

All in all it’s really clever, remarkably flexible and hellish to unravel.

But, once you know it’s there and how to find the keys you need, it makes a lot of
sense and is clearly very efficient, flexible, and powerful

Infinite Recursion in Parser Generators

2010-12-25T08:34:00.007-07:00

Well, I've stuck my foot in it again and doing something which doesn't make a lot of sense.

Skipping the details, I decided I need to write a parser in PHP and the language I'm designing is embedded in PHP - which has a complex syntax and . . . anyway, one thing led to another and I've ended up writing kind of a parser generator in PHP.

It's not really a parser generator - it's more like a programmable parser where the program is a grammar specification.

So, I broke out the Dragon book and started reading, built a programmable recursive descent parser framework object and a hand coded parser for language grammars so I can can program it and a programmable lexical scanner - all in PHP and it all works pretty well.

And then . . .

I couldn't solve my problem with it.

Why, you ask?

Well, the problem I have cannot be solved by a parse tree created from a right-recursive grammar - which is what the book says a recursive descent parser needs to process.

Why?

Because when a recursive descent parser hits a left recursive production (which is what I need for my problem) it goes into an infinitely deep recursion.

Why does it do that?

It's stupid.

It turns out that no only will simple productions like: a : a TERMINAL ; create infinite recursions, but various, well hidden, mutual recursions will as well.

So - having faith in the Book - I decided maybe I need something which handles left recursive grammars. So I read and read and thought and thought and - as usually happens - I got tired, went to bed, and woke up this morning with a realization:

"It's not the recursion dummy, it's because processing non-terminals don't eat tokens!!!!!"

If that doesn't mean much to you - that's OK. The rest of this post is a boring explanation of what's happening and how to fix it.

First of all - why isn't it obvious from the book? Because it's not in there because:

The book defines a mathematical formalism to describe language structure and parsing
Like good mathematicians, they then ignore the actual problem and get buried in the formalism. And then . . .
They come up with ways to solve problems in the formalism using programming techniques and computer constraints available at the time they are working
The 2nd, 3rd, etc generation of 'students' become teachers and so they just teach the formalism in the computing context of the time of the original work

My dragon book is copyright 1977. Torben Ægidius Mogensen's "Basics of Compiler Design" is copyright 2000 through 2010 [nicely written, by the way] and the syntax analysis is a rehash of the stuff in the Dragon book [to be fair, I didn't read it all, but this is true to the margin of error inherent with a quick skim]

Believe it or not, things have changed.

The Apple 2 computer didn't exist in 1977 (I don't think it did. I got mine in 1979 or 1980) and it maxed out a a whopping 64 Kilobytes of RAM [that's 1024 bytes]. The processor executed one instruction about every couple of microseconds. In other words, both memory and speed were very very limited, so a lot of work went into algorithm design - at the expense of clarity and simplicity of code.

As a result, the compiler generators tend to avoid recursion ["function calls are expensive and take a lot of RAM"], but rather tended towards memory and speed efficient algorithms. As a result, the compiler generator section of the Dragon book is heavy into table driven parsers using conventional, non-recursive, non-functional programming techniques.

And - finally getting to the point - they are so deep into formalism and computing environment, they never actually get to the point of "what causes infinite recursion in parsers".

Well, here's the answer: any algorithm which revisits the same non-terminal without consuming a terminal symbol will infinitely recurse.

Huh?

This highlights another problem in understanding compiler generation: the compiler-eze terminology stinks. It emphasizes the algorithms, not the problem we're trying to solve.

So, here's what the Parsing Problem is:

Given a string of characters, does it make sense in a specific, grammatical language?

OK - that's not specific enough to answer. So let's make it more concrete:

First we will define a bunch of things we will call words and symbols. A word will be a string of 'letters' without any spaces in them. In English we also allow hyphens, so 'cat-food' could be classified as a word. In PHP a word might be a reserved word - 'foreach' or 'if' - or something like that. Anyway, we decide how to find these things in a string of characters.

We're going to call the things we find 'tokens' and it's the job of the 'lexical analyzer' to eat the string of characters and spit out an ordered list of 'tokens'.

These tokens are what the Language Grammarians call 'terminals' or 'terminal symbols'.

I'd rather call them 'tokens' or 'words' because that puts the focus back on what they are in the language. The term 'terminal' puts the focus on the activity of the parser - which we haven't gotten to yet.

Now, you might try to build a grammar description using only 'tokens', but it would get pretty large pretty fast and it would be really limited.

So you need something else. You need things which represent parts of the language. For example, you might need something called a 'sentence' [starts with a capitalized word and ends in a terminating punctuation mark: . or ! or ?] and maybe a 'paragraph' and maybe . . . well you get the idea.

These things which represent parts of the language can be composed of tokens or other parts of languages. In fact, in order to be really useful, these parts need to be able to refer to themselves as part of their definition - that is 'be recursively defined'.

For example, lets say I have only four words: A, B, C, and D. I also have a couple of symbols, say AND and OR. That's my whole vocabulary.

Now let's say I want to construct sentences. I might say something like:

sentence : A AND B | A OR B | C | D ;

where I'm using ':' to mean 'defined as' and '|' as 'or it might be' and ';' for 'that's all folks'.

But this is kind of limiting. So let's say I want to build more sentences than I can list using only words and symbols.

word : A | B | C | D;

sentence : sentence AND sentence | sentence OR sentence | word ;

In compiler-eze, these parts of sentences are called 'non-terminals' - again, putting the emphasis on the process of parsing [the parser can't stop on a non-terminal] rather than on the structure of the language. I'm going to call them 'fragments'.

Now, there are two ways I can use a grammar:

I can build sentences using it - which you do all the time: writing, speaking, creating programs, etc.
I can transform strings of characters (or sounds) into sentences so I can understand them - this is called 'parsing'

Before we get to parsing, let's look at how we can use the grammar to create a sentence.

Let's say I want to build a sentence - but I really don't care what it means, only that's it's grammatically correct.

I'll start with the fragment sentence. But this doesn't get me a string of characters. Grammars can only build sequences of 'fragments' and 'tokens'. Tokens are made up of sequences of characters - which is what I want - but 'fragments' aren't: they are made up of 'fragments' and 'tokens'.

So, in order to build a character string - or say something in the language - I have to get rid of all the 'fragments' so that I have a string of 'tokens' which I can (at least theoretically) feed to the un-lexical un-scanner which will produce a string of characters - which I can then print in a book.

So how do I proceed? (the arrow (->) means 'replace a 'fragment' on the left with one of the alternatives of the right side of the definition of the 'fragment' in the sequence and write it on the right side of the arrow.) (which is easier to do than say)

sentence -> sentence AND sentence -> A AND sentence -> A AND D

and now I'm done. I have 'produced' a sequence of 'tokens' [TERMINALS in compiler-eze]

which I can un-lexical analyze to produce a sequence of characters.

Now in compiler-eze, the alternatives on the right side of the definition of 'sentence' are called 'productions', because replacing a 'fragment' by one of them 'produces' something which is grammatically correct.

Ok - this is pretty straight forward, if boring. So let's turn to the 'parsing'. That is, given a string of characters, is it a grammatically correct sentence?

The mathematicians would say 'it's grammatically correct if (and only if) there is a sequence of replacement operations I can find using productions which will generate the sentence'. So - as they would have it - they have 'reduced' the problem of 'parsing' to finding a sequence of productions which will produce the sentence.

How do we do that? The Dragon book starts by analyzing algorithms, but let's take a different approach: let's look at what we do when 'parsing' a sentence somebody says or that we've just read.

What I think you do (or we do) is look over the sentence and divide it up into chunks which make some sort of sense. Like 'Joe ran through the forrest'. Well, what's this about? 'Joe'

What did he do? 'ran' Where did he do it? 'through the forrest'. Stuff like that.

Let's formalize this procedure:

First we'll lexically analyze the sentence: for 'Joe ...' this amounts to classifying each word according to its possible uses:

'Joe' - is a noun and a name. It can be used in a subject or the object of a phrase
'ran' - is a verb. It can be used as a 'verb', as part of a predicate, part of a compound verb, or a phase ['seen to run']
etc

Then we start parsing by examining the first token: Joe. Some sentences start with a noun, so we put 'Joe' on the shelf and look at the next word to see if it fits with how sentences which start with nouns are constructed. etc.

The point is, we are scanning from left to right and trying sentence forms to see if they fit

the noise we just heard or read. [left to right, right to left, up to down - doesn't matter so much as the fact that it's really focusing on one word at a time in a consistent order].

So, in parsing we have two scans going on:

we are scanning the token stream
we are also scanning across a production to see if these tokens fit into it

The 'parse' terminates when the token stream is exhausted and all the tokens have been stuffed into 'fragments' OR something won't fit into any fragment. This is controlled by the sequence of scans across productions. Each time we start scanning, we start with some 'fragment' definition and exhaustively try all of it's productions to find a fit with the token stream - remember that we are scanning the stream left to right. So the only way to get into an infinite recursion is to find a production scan which does not terminate.

Scanning a production terminates on one of three ways:

a segment of the token stream matches the entire production - then the production is accepted. Accepting means that we don't have to look at those tokens any more and we can make a record of the fragment we recognized. [in compiler-eze we then 'reduce' by replacing the production by it's non-terminal in the non-terminal definition (again, emphasis on algorithm rather than process]
a token doesn't fit, in which case the production is rejected.
the production can be empty - and so it's trivially satisfied. [I forgot this early and have to think some more about it. Golly! that's meat for another post on this topic]

So if - in our scan across the production - we never look at any 'tokens', we will never terminate the scan. How can this happen?

Here's an artificial example:

frag1 : frag2 | WORD1 ;

frag2 : frag1 | WORD2 ;

No matter what I scan, my production scan will first look for frag2 which will look for frag1 which will look for frag2 which will . . . and I will never examine a token, so I will never reach the end of the token stream.

To go to a less artificial example, let's go back to my A, B, C, D language.

I'm given with a sentence A AND C and I want to see if it can be produced by the grammar. I decide to 'run the grammar backward' to see if I can find a sequence of substitutions which work.

OK, I start by guessing it's a 'sentence', so I write down:

sentence

Now I say - 'what production might this be? Let's try the first one!', so I grab:

sentence AND sentence

Now you can look at the whole sentence and say 'Yep!!! It fits', but the computer will only look at what it's programmed to do. So, lets say that I've programmed up a recursive descent parser, which works by defining a function for each 'fragment' which it calls when it sees it's name in a production.

So my 'parser' will see 'sentence' and call the 'sentence' function which will then look at the first production and will see sentence and will call the 'sentence' function and . . .

And there you are - infinite recursion.

So we can't use a recursive descent parer. Right? Well, . . .

The recursion isn't caused by the parsing method, it's caused by any algorithm which attempts to match the same 'fragment' twice without recognizing and moving past a 'token'.

So infinite recursion in parsing results from designing an algorithm (any algorithm) which can cycle through a sequence of 'fragments' without ever recognizing (and using) a 'token'.

So, can I patch up a 'recursive descent parser' so that it handles 'right recursion' and other forms of infinite recursion?

Sure. I just have to keep track of my progress through the token stream and reject any production in which a 'fragment' occurs which I'm in the (recursive) process of examining AND which is at the same place in the token stream as it was before. Again, this will be easier to code than to write.

I'll post a note when I've finished fixing this thing - in case you want to look at the code

Mike

Change - again

2010-08-02T06:41:00.005-06:00

Just about everyone I know would rather die than change something they believe.

That's too vague.

Let's say I believe I'm too fat. That can make sense if I look in the mirror and see somebody who looks like a sphere. But for an anorexic, when they look in the mirror they see somebody who looks like a stick.

The reason we say they are 'anorexic' isn't because they look like a stick. It's because they look like a stick and think that they are too fat AND they won't change what they believe.

So how do we react to this?

We call something like this a 'disease' and look for something to do to them to make them change. Probably some chemical we can put in a pill or an injection or a patch or a suppository.

Does it really make sense that an inert chemical can cause someone to have a specific idea? Isn't an 'idea' or a 'belief' more complex and specific than a single chemical?

So what can these 'drugs' really do? - other than slow down or speed up thinking?

If that's all they can do, then 'drugging' people just changes their ability to think - their 'thinking environment' - not their beliefs.

So 'drugs' can't 'cure their disease', although they may make it possible for them to think about it differently. Maybe they it makes them think more ssssllloooowwwwwlllllyyyyy. Maybe it makes them stop thinking at all. Or maybe it just makes them passive so we don't have to think about them at all. Or maybe - as my friend who knows these things says - they generally don't work.

But that's not the point.

The point is: if an anorexic didn't believe he/she was fat, she wouldn't be an anorexic. She'd be a skinny person who knew she was too skinny and would do something about it - like eat some more.

So how do you change a belief?

Take football for example. The team which wins consistently believes that they can win. Not only that, they believe they can win this game. Right now. If they think they can't, they always lose.

What makes them believe this?

It's pretty simple: they have a slogan, a mantra, a rallying cry, a whatever to repeat over and over again. So as long as they can keep telling to themselves they can win, they will win, they're going to win - then they believe they can, will and are going to win.

Is a belief anything more than something we keep repeating to ourselves?

What happens when we stop talking to ourselves about one specific belief? Doesn't the alcoholic or a smoker keep reminding himself that he needs a drink or a cigarette? What would happen if he - instead - reminded himself that he needs an ice cream cone? (Besides getting fat and maybe getting diabetes) Wouldn't he eventually go from being an alcoholic to an ice cream-aholic?

A belief is just a thought. It's not made out of stone or steel or even jello. It's 'mind stuff'. There's two kinds of 'mind stuff'. There are memories and there's 'what I'm thinking now'.

All you can do with a memory is either lose it or drag it out to 'think about it now'. Everything you do and experience is the 'what I'm thinking now' stuff. That's where the anorexic and the alcoholic and the smoker 'belief' exists.

There isn't any automated thought loader which pushes thoughts into your 'thinker' and makes you think them. You get to pick and choose.

Don't believe it? Close your eyes and try to count the thoughts which come up over the next 10 seconds.

If you're like I am, there were a lot of them. Ten, a hundred, I don't know. Just lots and lots of them. I'll bet you 'thought about' just a couple - maybe one or two. What happened to the rest of them? They're like the kids you didn't pick to be on your team: they just wandered off.

The stuff you and I believe - about life, goodness, and - especially - ourselves - are just these familiar little thoughts we keep repeating. And by repeating them, we think their real. And that's all a belief is.

So really, how hard is it to change a belief?

It's easy - if you want to and are brave enough to give it up.

Human Development - kind of

2010-06-20T13:02:00.004-06:00

So somehow I went from wherever I was as a young guy to this worn out husk who had to deal with with raising six kids. Was nothing I planned to do, but, well, there it is.

That's not the point - it's just to give you my credentials, assert my authority, bolster my claims, etc etc.

This first point is that I'm writing from my experience - not text book theory or some other academic point of view [I have those also, but not about kids]

Here's the Second Point:

When a kid is just born, they don't do a lot and the way they interact is really different from how they are later on. They're kind of like little animals that just crawl or run around. They don't really argue about anything. Their complaints are very personal and immediate - hungry, wet, cold, lonesome, tired.

After a couple of years something happens. You've been going along saying things like: Please don't do that or Please do this - and almost getting used to the fact that they never really pay much attention and don't seem to remember what you said from one second to the next. But then the thing happens: the kid turns around and looks you straight in the eye and says - very firmly and usually pretty loudly - "NO"!

This is a major breakthrough.

The kid hasn't turned into a monster or become a 'Terrible Two' (as I was told by my mother, over and over again). The kid just now finally noticed that you're trying to get him (or her) to do something that wasn't what he wanted to do.

That's the point when you can finally start teaching the kid how to 'not get run over by a car' and 'to clean your plate' and 'to pick up your toys'. It won't work - at least not for the first 10 or 20 years - but you can now get started and be (relatively) happy with the knowledge that you're not just being ignored. You're being Actively ignored and - believe it or not - all of you talking, pleading, reasoning, and (most of all) the example you set is sinking into that little mind.

After mulling this over for a bunch of years - I've come to the conclusion that this is a pivotal point in all of our developments. It's the point where we realize that we live in a context over which we do not have total control and with which we must learn to interact.

And for a Third Point:

Lora (we're married and four of the kids are her fault) is a certified Montessori method pre-primary instructor. She went to school for a year, did a year internship, read all of Montessori's books and practiced on our kids. Turns out that's enough for her - she prefers ponies and dogs to humans.

Anyway, she has creds as well.

Montessori discovered - by observing kids - that kids go through various phases of development where they are really different. She called them sensitive periods and it's fascinating stuff.

But - to get to the point - there's one thing I wanted to write about here. It's the thing which happens when the fish in the classroom fish tank dies.

The pre-primary Montessori classes have kids in the 3 to 5/6 year range. They generally have a fish tank. At some time during the year one or more of the fish will die.

Here's where it gets interesting.

The kids come in and quickly divide into two groups.

The younger kids look at the fish and say 'the fish is dead' and then go do something.

The older kids look at the fish and say 'the fish is dead. I wonder why the fish died. Was the water too hot? Was it too cold? Did another fish kill it? Wasn't the food right? I think that ...'

You get the difference?

Before something changes, it's ok to just see the dead fish and register it as a fact.

After 'the change' we have to 'know Why'.

I think this is the origin of Why.

You should listen to some of the silly 'reasons' 5 and 6 year old kids come up with. Their logic is not all that bad, but the 'facts' they start from are pretty lame. But that makes sense because the don't have much experience.

So do that for a while - - - and then - if you're brave - listen to some of your friends and neighbors talking about stuff. Things like 'the problem with ... is ... because ...'

When I started listening to adults and comparing it with the stuff 5 and 6 year olds come up with, I got really upset. It's the same stuff!!!

I think everybody is acting like 6 year olds - including the guys who run the countries and big companies.

Doesn't that explain a lot of what's going on?

I'm telling you - the problem is the whole thing is being run by 6 year olds and that's why nothing works. The answer is . . .

ORM or Not? Part Two - Definitely Not

2010-06-20T12:19:00.006-06:00

Maybe that's a little too harsh - but I don't think so.

Here's the background:

One of the biggest recurring programming problems I've had to deal with in site and application development has been - (drum roll) - developing the database. Not populating it - that's just boring. Developing and refining the data definitions.

The "Conventional Wisdom" promulgated by Software Gurus is basically this:

Databases are Good With Data
Data Objects Are Good With Data
A Programmer Must Define a Mapping Between these Two Good Things and then All Will Be Good

Not my experience.

Software Gurus don't write application. Software Gurus write Books and (occasionally) toy examples. So they really don't understand that Good does not generalize because the Good that Databases are with data is all about safety and accessing it in great and small piles. The Good that Data Objects are about is manipulation, fine structure, and flexibility.

My experience with Object Relational Data Mapping - which is just a fancy phrase for how you get the data from the database into an object and back - is this: when I Did it Their Way, I had to hand-coordinate both my Objects and my Database.

So when I wanted to add a field or change a field or change a datatype, I had to do everything twice.

Now Rails has Active Record - which is a stupid Software Guru name for an ORM which creates the Data Objects automatically (my drivers license is an active record - meaning it's current so I can legally drive), but that doesn't 'solve the problem' it just moves it. [maybe Rails 3 is smarter,
I dropped out just as Rails 2 was coming out]

Specifically, the Rails solution moved the problem to 'migrations' which turned out to be fragile. I know: I tried it.

So, a couple of years ago I proposed Not building an ORM, but rather loosening the coupling between the Database and the Data Objects.

Here's what I've done:

Defined an PHP class which implements a data abstraction which includes most of the types of data and methods I want in my CMS [see http://www.yasitekit.org] and knows how to map those data types into database fields. It also knows how to create the database structures, create, update, delete, and retrieve those values.
Defined a PHP class which makes it easy (relatively speaking) to create objects which are instances of the data abstraction class. These classes come with a bunch of methods which I've found useful as well as a management class which automatically provides interactive data administration.
Some automated analysis tools which work with this stuff so it's possible - and relatively painless and safe - to add, delete, and modify the derived objects (point 2) and reload the database. All done by hacking the derived PHP objects, but never ever touching SQL.
As a side benefit, I threw together a database adaptor which abstracts the 8 essential database operations [create/destroy database;create/drop table;insert,update,delete,select] at a level high enough that we never have to muck with SQL. This makes it possible to augment it with non-SQL databases - such as Mongodb. In it's present form it allows painless migration between sqlite, mysql, and postgresql (it currently handles 5 different PHP database interfaces - automatically figuring out what's available - yada yada yada)

So, at this point, I feel more than comfortable saying No to ORM's.

BTW - if you go to the YASiteKit site, you may be a little dissapointed because all of this good stuff is not available yet. But It Will Be - Real Soon Now (no kidding). And a whole lot more.

What is on the site is a pretty-much full set of documentation - both about the system design and about how each of the parts work [I write doc as I go, so it's not all pretty - but it's useful (I know because I use it)].

So take a look. If you have any comments - let me know.

Also, if you like it enough to want to help - let me know sooner - I'm almost ready to turn it loose to see if it gets any traction.

Choices [Rant Warning]

2010-06-01T23:03:00.006-06:00

I've been thinking a lot about the idea of Choice lately. Not so much any particular choice, but the whole idea itself and how it differentiates us from rocks and water and stuff like that.

It seems to me that one of the distinguishing features of 'life' verses 'non-life' things is that 'living' things get to choose what they do rather than just follow the 'laws of nature'. For example, rocks don't do much of anything except just sit where they are or roll down hill or get knocked someplace else by something else, but a bug can get up and go someplace else - even against gravity, if it wants to.

So what? Well, I've been wondering for a long time why we (that is us 'humans') do such stupid things and - even when we realize they are stupid and we are hurting ourselves - keep doing them. We even go so far as to tell ourselves - and everyone who will listen - how hard it is for us to not do them.

Take eating too much. When I eat too much for a while, I get fat and crabby. Then I feel sorry for myself because (I claim) it's oh, so hard to not eat too much. It sounds like there is an army of bad guys stuffing food into my mouth - but what's really happening is that I'm choosing to pig out. It takes real effort to keep eating when I'm full or when I'm not really hungry.

That one's simple and easy to see. And it's easy to see that if I'm fat, I made myself that way and I did it because I wanted to eat more than I wanted to feel good.

But it goes a lot further than just eating.

I think we've made up all kinds of religious garbage to excuse our disgusting behavior. We make up and believe in gods and devils and genetic forces and team loyalty and goals and success and all kinds of other junk to take the heat off of ourselves. 'The Devil tempted me - and I was weak',
'It's in my Genes - I can't help myself', 'We've got to do it for the Team', ' .. or the Company', '... or our Kids', '... or for Honor', '... or Because That's the Kind of Man/Woman I Am'. etc

It's all a smoke screen.

We do it because that's what we choose to do - just like those guys who strap bombs on themselves and kill a bunch of people.

It's a personal, misguided decision.

The hell of it is, it isn't hard to Stop. It doesn't take Effort to not do something. There really isn't any force making us do this stuff. Want proof? Just anethesize your mind somehow - and you won't be doing all that stuff you think is so hard to not do.

So let's get over it and stop kidding ourselves.

We muck things up, piss each other off, get mad about stuff, drill oil wells places where we can't plug leaks, kill people, destroy the whole bloody planet and everything on it because we choose to.

That's all there is to it.

And that's what makes us different from a nice, peaceful rock.

Understanding

2010-05-06T15:10:00.007-06:00

To Understand, to Have Understanding, to Know

What does it mean when we say "I understand"?

Like everybody else - or at least I think everybody else - I used to take it for granted. I thought that I knew what it meant.

Now that I'm getting to know more about what goes on in whatever it is I call my mind, I'm pretty sure I was wrong.

Here's what I've figured out so far.

Thinking

First of all, we seem to think that we think.

Some of us think we think "rationally", but most of us have no idea what that actually means.

It seems to me that we - as a species - have been working at understanding what 'thinking' is for a very long time. At least since the Greek philosophers and the Chinese sages. Probably longer than that - at least 2,500 years or more.

Somehow we've latched on to 'logic' as 'real thinking' and everything else as some sort of minor annoyance.

I don't agree that 'logic' and 'rational thinking' are the real kings and queens of all 'mental processes' and that the rest of whatever goes on in our minds is at most second rate.

And here's why:

Logic - a Necessary Digression

We've studied the hell out of Logic. We've formalized it and we know what it is. Well, those of us who've taken the trouble to study it a little know what it is.

First, what it isn't.

Logic is not sticking 'because' in the middle of sentences. It's not making up 'reasons' for things. And it's not asking and then answering 'why'?

Logic is a rigorous, formal method of evaluating the Truth or Falseness of sentences which are constructed according to specific rules.

First of all, what is a sentence?

In 'Logic' it's a string of symbols consisting of 'propositions' and 'logical connectives'. The 'propositions' are just blobs or words that can have any structure and may or may not mean anything. For the logician, the only thing that matters is that they are either True or False. In fact, for the logician, it doesn't matter at all what True and False mean - only that they are different and only that there are only two possibilities.

The 'logical connectives' are special words which can be used to 'connect' two propositions or sentences. When stuck in between two of these things, the three of them form a new 'sentence'. That 'sentence' is either True or False and the value strictly depends on (1) the values of the two things on each side of the connective and (2) the rules of the connective.

Let's get formal:

P, Q, and R are all propositions or sentences - which means they have truth values.
* AND, OR, and IMPLIES are connectives
Let's add in NOT, which isn't a connective, but it's useful. It's 'logical negation' - which means that if P is True, thenNOT P is False - and vice versa.

So we build sentences by writing P AND Q, Q OR R, P IMPLIES R and things like that. We evaluate these sentences according to the rules:

if P and Q are both True, then P AND Q is True, otherwise it's False
if P and Q are both False, then P OR Q is False, otherwise it's True
if P is True and Q is True or if P is False, then P IMPLIES Q is True, otherwise it's False
if P is True, then NOT P is False and if P is False, then NOT P is True.

If we add in some Parentheses, we can get really wild and use sentences for propositions and write things like:

((P AND Q) OR ((NOT R) OR S)) IMPLIES Z

We could also write the same thing without the parentheses, but we wouldn't know what it means without using some special rules. Why? Because we wouldn't know if

P AND Q OR NOT R OR S IMPLIES Z

means (P AND (Q OR (NOT R)) ) OR (S IMPLIES Z) or the thing I wrote above.

But that's for really studying 'Propositional Logic' - which you can find in a book. For now, it's not really important.

The important thing is this:

The Truth of the sentence depends completely on the Truth of the basic propositions - that is, the propositions which don't contain any 'logical connectives'. These are the 'Axioms'. It's what you start from.

Everything you write using Logic can be traced back to and completely depends on the Axioms - the Propositions you write down and start from.

This means that Logic only transforms the shape of the original propositions. It can't add anything new.

Working with Logic is like walking around a house and looking at it from different sides. No matter what you do, it's still the same house. Nothing is ever added or subtracted from what it was to start with.

There's more Logic - for example, First Order logic adds 'quantifiers' to the Propositional Logic we've just outline. It adds exactly two: the Universal Quantifier and the Existential Quantifier.

This lets us write richer sentences because we can then write things like: All P is True and At Least One P is True.

Having quantifiers makes is easy to tell when someone is saying something stupid. For example, if somebody says "All cats are mean", you can tell they don't know what they are talking about if you've ever met a cat that wasn't. So you object, and then they say you're being picky and that they really meant "most cats are mean". That is supposed to make it better, but it really
means that they don't really know, but want to think of it that way. So, it's better to keep your mouth shut and just know that they say stupid things.

Also, it helps tell what you can know for sure. For example, when somebody says "you can't do that", but you think it would be a good idea to "do that", all you need to do to prove that
it's possible to "do that" is to find one example where it worked. Then you can ignore them. This is a great help when starting a business, because most everyone will tell you it "won't work", but if you can find an example of when "it worked", you can ignore them because you know it's possible.

There are a lot of things like that.

So Why Logic?

Well, if I want to do something AND I can figure out enough things which are True, then I can use Logic to figure out if the thing I want to do will work.

Or, I can take the thing I want to do and I can see if it's Logically Consistent with a bunch of other things that I have to do or something like that.

Or maybe the thing I want to do has to have something which I can't do. If I can figure that out, then I can avoid trying to do something which won't work.

Logic can keep us out of trouble. It can help us predict if something will work or if it won't.

Knowing things like that saves a lot of time, money, anguish, and other things we want to save. It helps us be successful - whatever that means to each of us.

Logic is reliable.

Why is logic so reliable?

Not magic: Logic is a bunch of rules for figuring out if things will work based on thousands of years of experience.

So Logic is the Answer!

Right?

Wrong!

Propositions

Where do the Propositions come from?

Remember, the Logic just transforms our propositions - our axioms - our guesses of what is right or wrong.

If our Propositions are crap - then all the Logical Reasoning in the world won't make them smell good. It will only be looking at the crap from different points of view.

Crap is still Crap.

One place our Propositions don't come from is 'reason' or 'logic'. We get them by following some other rules.

Scientific Propositions

One set of rules are the ones used in Science: a fact is an independently, verifiable experimental result.

Let's take that apart:

a result is something which can be measured. This means that there is a mechanical method which transforms something which happens into a number. The mechanical method must be reliable. For example, the diameter of an object. If it's a hard sphere, then that's easy to do; if it's a rectangular solid, then we need more rules (for example, measure each width on a line normal to each face and compute the average of the three measurements); if it's a bag of gas, then we're screwed because we can't reliably measure the diameter.
an experimental result is a _result_ measured from an activity which can be described precisely. The precision must be sufficient to compute the accuracy with which measurements
a verifiable experiment is one which can be performed again. In order to do this, the activity of the experiment must be completely described in enough detail that the activity can be performed with the same precision.
an independent experiment is an experiment performed by a different experimenter, using different equipment at a different time and place from the original experiment. This depends on the precision and completeness of the description and removes any bias the experimenter, location, and time of the first experiment may have introduced into the results.

This is pretty restrictive. It's also pretty slow and pretty expensive. It's also pretty good at building reliable knowledge.

So that's one way of getting propositions. We design experiments and do them to see what happens. We verify them to make sure that we know what to expect. Then we try to figure out rules which describe what we've observed and then test them using Logic to look at the results from different angles - so to speak.

How else can we come up with a Proposition?

Guesses

How about we just guess?

Guessing is good. We do it all the time. Most of the time it doesn't work though.

But often, it's all we have.

Say it's election time and you decide to vote. Who are you going to vote for? You know both of them want to get elected and that both of them will say about anything they think you want to hear. In other words, most of what they say are lies. So you guess. You say "I think this one is more likely to do what I want" and then you pull the vote lever.

Now if you were being "logical", you'd do something else because you know both of them are lying. So maybe you wouldn't waste your time voting. Maybe you'd logic yourself into making a lot of money so you could just bribe whoever won. That would be more 'rational', if you want to get a politician to do what you want.

Emotional Propositions

How about we just claim something we want to be trueactually is?

As far as 'Logic' and 'Reason' goes, this is just fine.Remember, 'Logic' starts after you've got the propositions - it doesn't care where you found them. And inasmuch as 'Logic' is formalized 'Reason', well, the same can be said for 'Reason' as well.

When we do that and use 'Logic', we call that 'Rationalizing'.

We're 'just making up reasons for what we want to do'.

This is how a lot of real disasters are created.

Take starting a war.

Does it really make sense to rip everything apart, destroy somebody else's hard work, their lives, etc etc?

Sure - if you're willing to cause that much pain and you want their stuff.

Or maybe you think you need to think they're evil and that you're so right that you have to stamp out the evil.

But enough of this

Back to Thinking

I think you'll agree that - looked at this way - rational thought isn't really much 'higher' than any other form of 'thought'.

We're really just kidding ourselves.

So what is all this 'thinking' and 'understanding'.

What I 'think' about 'understanding'

When I'm really honest with myself, I say that I understand something when I have a feeling of comfort and confidence that I know what that 'something' will be like the next time it comes up.

If it's a freight train moving along - I 'understand' that it will stay on the tracks and I can control whether or not it squashes me.

Of if I'm trying to sell somebody something - I 'understand' that if I get the price right and am patient enough and advertise it enough, somebody will come along and buy it.

Sounds pretty good - doesn't it.

What Happens if things happen like I Predict?

When things happen the way I predict - then I think I'm pretty smart, I get more confident, and I 'lean on' my 'understanding' even more.

Does that mean that I can make accurate predictions?

Experience says: Well, Maybe. It all depends.

What Happens if things don't happen like I expect?

Well, lots of things.

Mostly I used to ask Why?

Then I'd come up with some Propositions to use to build a logical argument explaining how what I understood should have happened, but didn't.

This would usually involve finding somebody to blame. (If they'd only listen to me or do the right thing or weren't so self centered or . . .)

Then I'd can feel comfortable again because I'd 'know why it happened that way'.

In other words, I'd 'understand'.

(I know you would never do anything like this. You understand things a lot better than I do - don't you?)

And So, . . .

It's just one big circle of self delusion.

If you believe this stuff, you probably feel uncomfortable.

So even if you know it's true, you won't feel that you 'understand' it and will want to try.

See the trap?

Timing Tests

2010-04-23T09:59:00.004-06:00

OK - I'm not a timing test expert. I'm not a program profiling expert. etc etc etc

I know timing tests are 'hard to do right'. I know that there are all kinds of consideration. I know that 'to do it right, you have to . . .'

But I don't really care about 'doing it "right"' according to some picky standard.

What I do care about is not writing really slow code.

For that, the rules are simple:

Rule 1: Don't do things that take a long time

Rule 2: If you have to do repeat something a lot, check out alternative ways to do it and pick the method which is both clear to read / understand and takes the least time.

Rule 1 - expanded:

You do this by knowing how long things take and which operations are blocking. You don't need to be accurate, because for most things, 'takes a long time' is measured in orders of magnitude.

Here are the relevant cases:

Monolithic program doing in-memory data access/processing
Multi-threading/parallel processing/whatever - any form of parallel processing which is executed inside a single process context. Here you tend to lose because of blocking and communication - one thread needs to access shared data and so blocks reads, etc OR one thread needs results from another to continue OR etc.
Self generating code - aka Metaprogramming. This is a cool way to impress your friends, but it costs orders of magnitude in performance. The idea is to write code which traps function calls to functions which don't exist, then parse the function name and build a function 'on the fly' to do the task encoded in the function name, dynamically build the call sequence, execute the function and return the result. It's not hard to do, but it's pretty much unnecessary (almost all the time) and really slows things down, because parsing strings takes a lot of repetitive work. That's why we have compilers!
Disk reads are much longer - at a minimum they require a context switch as you make a system call. Then it depends on file size, caching in the operating system, memory size, etc. The Rule is: for repeated reads, try to do only Once and cache the result in a variable
Run a subprocess - this requires a context switch, process invocation, lots of disk reads, etc etc followed by receiving result, parsing it, etc etc. Much more expensive than disk, but less expensive than Network reads.
Network reads are longest - not only do they require a system call, you typically have to run another process someplace. If that process is on a different host, then the cost is astronomical relative to in-memory and disk i/o.

So Rule 1, says - if you don't really need to do the Slow Thing, then don't. And Do the Slow Thing as seldom as you can get away with.

Rule 2 - expanded

Right now I'm writing a lot of PHP (don't groan, it seemed like a good idea at the time) and in this code I have lots of places where I need to do things based on the value of a string. For example, I'm writing a lot of PHP5 objects where I put guards on attribute access so that I can find spelling errors (my High School English teachers understand why I need to do this).

So I have lots of functions that look like:


function __get($name) {
if (in_array($name, array('foo', 'bar', 'baz'))) {
  return $this->$name;
}else {
  throw new Exception("$name is not a valid attribute name");
}
}

function __get($name) {
if (($name == 'foo' || $name == 'bar' || $name == 'baz'))) {
  return $this->$name;
}else {
  throw new Exception("$name is not a valid attribute name");
}
}

function __get($name) {
switch ($name) {
  case 'foo':
  case 'bar':
  case  'baz':
    return $this->$name;
  default:
    throw new Exception("$name is not a valid attribute name");
}
}

I do this a lot, so I need to know which one is the fastest. I don't need to know precisely, I just need to know 'more or less'.

To do this, I need to build a test case and run it to get some timing numbers.

The test case doesn't have to be perfect, but it does need to put the emphasis on the differences between the three different methods. It also has to be large enough to be able to distinguish run times between the methods.

In this case, I built the three functions each with about 150 alternatives and the built a list of trials which would fail about 1/2 the time. I then executed each function a bunch of times.

How many is the right bunch? I'm lazy, so I start small for the number of repetitions and then crank it up until the total run time per method is around 10 to 60 seconds.

Here's what I got:

switch: 49.7731 seconds
in_array method: 86.3004 seconds
if with complex conditional: 57.0134 seconds

Guess which method I'm going with.

[guess how I'm going to refactor a lot of my code (sigh - I should have tested first)

Self Image, Self Identity and All That

2010-04-19T14:11:00.008-06:00

Who am I?

Or, more to the point, what is the 'idea' of myself that I identify with?

Or, even more to the point, how do I 'like' to think about myself?

I put 'like' in quote marks because 'like' doesn't necessarily mean 'enjoy' or 'makes me happy', but here it means 'what I keep coming back to because I believe it's true'.

In other words, the way I 'like' to think about myself might not be very nice - if I'm convinced I don't measure up to my ideals.

Everybody 'thinks' of themselves as something - has an expectation of who and what they are. In other words, Everybody 'likes' to think of themselves in a particular way.

That's the setup. Now, here are some questions:

Am I nothing more than an opinion? Or am I real?
Can I change 'Who I am' by changing my opinion?
Is my 'World View' a result of 'Who I am' or is it something I create for my 'Who I am' to live in?
Do I see and hear the world around me OR do I pick and choose what I hear and know?
Do I really know 'Who I am'?
Do I really know my friends? Or do I make them supporting actors for myself?
Can I live without knowing 'Who I am'?
How can I avoid living in an Illusion if I continue to 'know' 'Who I am'?

First Hypothetical

Let's suppose that 'Who I am' is an opinion.

Opinions are just ideas that can be changed. They aren't 'facts'.

If I hold one opinion today and another one tomorrow, it's unlikely I will be arrested, burst into flame, or that anything else substantial might happen.

I'll just have a different 'opinion'.

So, with my different opinion, won't the World be different?

Won't my friends become different people?

Won't the boundaries between good and bad and Right and Wrong shift? Won't they have changed just enough so I can make my 'opinion' work - at least as well as my old one did?

All I need do to test this is genuinely change my opinion once and see if this is what happens.

If it works like this, doesn't this mean that 'Who I am' is an opinion? An Illusion? and that I am living in a false world of my own creation?

Do I want to know?

Second Hypothetical

Let's assume 'Who I am' is somehow 'real'. It doesn't matter what this means other than that it is something other than an opinion which can be changed at a whim.

Now one part of my World View ranks everything by how 'good' and how 'bad' it is. There is usually a sliding scale from 'good' to 'bad' with 'saintly' on one end and 'absolute evil' on the other.

Naturally, I will think of myself as more 'good' than 'bad' - no matter how I think about how I live up to my expectations. [for example, if I think of myself as falling far short, then I will still think of myself as 'better' for having noticed this and for admitting it to myself]

So how will this effect my World View? How will I tend to filter and interpret that which I see?

Isn't is natural for me look for the evil and bad - so I am - in contrast - much better 'than average'?

Won't I go out of my way to do so? Won't I respond with much satisfied emotion to my discoveries of the evil in others? Satisfied in my own 'goodness by contrast'?

How can we test this?

Isn't this consistent with both the continuous litany of complaint and criticism - in the press, in entertainment, and in our own, wool gathering minds?

What happens if I see lots of goodness around me? Doesn't that push my 'Who I am' down into the muck of badness - or at least shift me down a little?

If I can't change my 'Who I am', then I will not 'like' myself (and remember 'like' means what I said it means up above). Isn't that hard to tolerate? 'Who do those "goody, goodies" think they are anyway?' Doesn't it seem natural for Cain to kill Abel?

Third Hypothetical

Again, suppose 'Who I am' is an opinion.

Then there must be something which has that opinion.

That something must be able to observe - inasmuch as it has thoughts, the 'opinion' being one of them.

So can this 'something' watch it's opinion and the thoughts it's opinion is thinking? (or maybe the thought's it is thinking for its opinion).

If this is true, then 'Who I am' is an opinion and the 'something' can become aware of this.

How can we test this?

Can we watch our own thoughts? As we think them?

If we can, then this is true and it opens the _possibility_ that the 'Who I am' is an opinion and that it can be changed.

If this is true, then can't psychic trauma be impermanent? And if impermanent, can't it dissipate? And if dissipated, hasn't it been healed?

Further, how is an opinion maintained? It isn't made of wood or metal. It has no substance other than thought. If thought isn't thinked, then is isn't. It's not there. It's gone.

So, if psychic trauma is thought, isn't it impermanent and has to be 'thinked' over and over again in order to be? So isn't not-thinking it the path to it's dissolution?

Is the dwelling on 'the bad things' and 'how sick I am' the cure or the cause of disease and despair?

Fourth Hypothetical

'Who I am' and 'Who You are' are different.

It doesn't matter if they are real or just opinions.

You see the world differently from how I see the world because you must shape your 'world' so it fits your 'Who I am' and I must do the same.

But mine is different from yours, so our 'worlds' are different.

Can I really see 'Who You are'?

Can I do more than guess?

Suppose your 'Who I am' world conflicts with my 'Who I am', from my point of view. Won't I filter and squash what I see and hear to fit my 'Who I am' instead of yours (no matter how honest, just, and polite I think I am)?

So how can I ever see where you don't make my 'Who I am' good and right? And can you see me?

Deep down don't you think you're a little better than me? I know I am a little better than you - or at least a little righter.

Doesn't that prove we can never know each other?

How can we converse?

Aren't we having two meaningless conversations with ourselves while we pretend that the other is there?

Fifth Hypothetical

I find that knowing 'Who I am' leads to an expectation that I will continue to be 'Who I am' and that I interpret and bend everything I see, hear, taste, smell, feel and think so that that will be true. I insist on continuing my existence as I envision it.

Doesn't this mean that I've been living in an illusion?

Can I escape the illusion without giving up this expectation, this prediction of the future?

Realizing this, can I continue to maintain the illusion - knowing it is a lie?

If I give up my expectation that I will continue to be 'Who I am', won't this mean I will change 'Who I am' into something else? And can I tolerate replacing one 'Who I am' with another?

News Flash: PHP Documenters Insane!!!!

2010-04-10T09:20:00.003-06:00

The PHP documentation has gone from very useful to hideously obstructive.

The people who are rearranging the doc into little, tiny chunks which are hyperlinked all over the place obviously never write code.

I just spent 10 minutes trying to find the name of an IO Exception so I can use it in some code I'm writing.

Old Doc:

I would go to the index, click on Exceptions and then scroll down the page (or do a find on IO) and there it would be. 10 seconds tops.

New Doc:

Go to the index click on Predefined Exceptions
Click on Exception - find description of Exception Object - info not there
Back Button
Click on Error Exception - find description of Generic ErrorExeption object
Back Button
Click on SPL Exceptions (what the hell is this? - something new?)
Look at Table of contents: 13 Exception Categories - none of which
looks like an IOException
Click on Predefined Exceptions in the See Also -
Back to Previous Useless Page - And Repeat

First they completely screw up the Perl Regular Expression page by chopping it into tiny, obscure chunks and now you destroy the exception documentation.

To the PHP Documentation Project:

PLEASE put it back the way it was.

Or get somebody who actually uses this stuff like a handbook while writing code to fix it

Or shoot somebody.

To Everybody Else:

Maybe the documentation people have stock in a book company and want the reference books to succeed by making the online Doc unusable?

All I can say is that the way they are going is really going to help Rails and Django.

What do you think?

P.S. Please Send a Nasty Note to the Gods of PHP

Why are our Programming Languages so Bad?

2010-01-28T21:17:00.004-07:00

I just took a quick look at Scala and Lua . . . and I don't think either one is a winner, but for different reasons.

The Scala guys seem to have fallen into the arcane syntax trap - with lots of critical information inferred. I think I agree with this blog post from January, 2008: Scala is probably not a readable language.

Scala seems partially motivated by the Java mistake of believing that 'more required words make more readable code'. That just makes the programs bulkier, not clearer. Other parts of the motivation seem to be to try to find a syntax which supports functional programming, OOP, and everything else which might be fun.

On the other hand, Lua doesn't have a rich enough structure. I think I'd rather write in C than something like Lua. Lua isn't OOP, it isn't a functional language, it doesn't even support structures sufficientlyl, let alone objects. I've had a lot experience working in awk - which is nice, but does not have good support for complex data objects, which makes the programs difficult to maintain and . . . this could get boring fast, so forget it. The bottom line is: languages which don't support a decent object model slow me down too much to bother with.

I've been writing a lot of Python lately - after burying myself in a PHP project for about a year and a quarter. I also write a lot of shell script, HTML, CSS, and whatever. I don't write Ruby anymore - but I don't want to get into that now. Before this I wrote a lot of C, Pascal, awk, etc and - believe it or not - about a half a ton of FORTRAN. So I've written a lot of code in some pretty awful languages.

The simple fact is that none of the modern languages are any good. They all suck.

Why is that?

We really know a lot about language design by now - or at least we should.

One of the things which really irritates me is that every damned language uses different syntax for the same semantic concept. I don't think I know of two languages which implement if ... else-if ... else ... the same way. [else-if is spelled 'elif', 'elsif', 'elseif', or 'else if' or doesn't exist].

It is now a fact that programmers work in multiple languages. These stupid, unnecessary syntax inconsistencies make life hell.

I'm starting a list of what should be in a modern language:

It should be syntactically as small and clear. Don't use words where symbols are clear: i.e. don't use 'begin' and 'end' - curly braces work just fine.
No reserved words. We don't need them. Somebody - I think it was Kernighan - pointed out that a decent parser can determine the meaning of a word based on the structure of the sentence, so that 'for' can mean one thing in one context and something else in another.
It has to support objects with (at least) single inheritance. I think Ruby got that mostly right. I think Python got it wrong by supporting multiple inheritance. I like Ruby's idea of Modules because it allows excellent code reuse. It gets us the utility inheritance promises without the head aches.
Scoping has to be correct from the start. Block scoping the way C does it is right. Matz got that wrong in Ruby - I hear they're changing it again in 1.9. Python still doesn't have it right yet, but I think it's getting closer.
Global variables are Evil. Crockford is right: They should not exist.
Variable declarations are a pain - but less painful than tracking down spelling errors which the language accepts. I've lost a lot of time needlessly hunting down misspelled words in PHP that would have been caught by simply requiring variable declarations so the compiler could catch them. So, variable declarations are Good.
String handling is Good. If you don't think it's a necessity - go write some string handling in C for a few years.
Dynamic - aka Duck - Typing is Good - we need it. I wasted too many years living without polymorphic stuff to ever go back [read: writing in C, pass a pointer and figure out what it is inside the function and hope to hell you don't screw up].
Static Typing is Good - we need it too. I've wasted too many years finding bugs which could be caught by a decent type system.
Functions need to be 1st class things. In fact, everything needs to be.
Closures are Good - we need them.
Functional programming is good - We need it.
Imperative Programming - Structured style [like Dykstra told us] is good - we need it too.
Operation overloading is Good - We need it. Everything should be overloadable. I think Ruby got that right as well. Python has been incrementally getting there for years.
Self modifying code can be a good thing - but it's hard to do right and rarely needed. I think it's better to support it directly rather than trying to discourage it. This is in spite of the crap the Ruby community loves to write [they call it meta-programming, but it's not really meta programming - it's automatically generated code] For some reason they think that self-modifying, self-generating code is intrinsically a good thing - but then I used to do a lot of stupid stuff when I was younger too. I think this is a result of lack of experience - especially in maintenance - and a lot of incompetence. It sure makes Rails a mess.
Exceptions are good - We need them. Error handling is always an issue and good, clean exception generation and handling support makes it easy to include it.
Interpreted is Good. It makes writing code much faster. Must have a REPL.
Compiled is Good. We always need speed. Only the stupid say 'speed doesn't matter'. It always matters, but very rarely at the expense of clarity.
Do we need an IDE? I don't think so, but I don't know. I just write in TextMate on my Mac. I used to write in Emacs - and still use vi from time to time. I've tried Eclipse, but just couldn't deal with it. I think this is a non-issue except that the Language should support development without and IDE.
Reflection - aka inspection - aka whatever - is Good. Python has that pretty right. All functions and classes take an optional documentation string which you can print in the interpreter by typing print foo.__doc__. It saves lots and lots of time paging through documentation. There is also a builtin function called dir() which generates a sorted list of all the attributes of it's argument. Ruby has that wrong - I don't know how many times I wrote foo.methods.sort to try to remember the name of a method when hacking Ruby.
Automatic Document Generation is Good - but the current systems stink. They are like a tail wagging a dog: Code is always more important that comments - and all documentation is comments. Why? Comments don't execute, so they always have bugs and eventually diverge from the meaning of the code [See Brooks: Mythical Man Month where he points out that it's not possible to keep to separately maintained files in sync] Documentation needs to be unobtrusive, compact, and easy. I have no idea how to do this - yet.
To be Continued

Any Things to add to the list?

Apple Non-Useability. Is Apple Copying Microsoft?

2010-01-12T19:15:00.003-07:00

I don't get it

Apple folks almost invented Usability testing and analysis. Tog, Nielsen, whoever else.

I just upgraded to Snow Leopard. It runs faster but . . .

- Preview defaults to pdf displays that are Too Big. I wasted 1/2 hour fiddling around finding controls. AND there's not little box that says what size the image is. Is that too much to ask?

- iTunes doesn't seem to be searchable. Looks like Apple folks are so enamored with graphics that they've forgotten that some of us might want to find what we want - Not what the Staff likes most or the newest or most popular. The browser thing is hidden in the View dropdown under 'Show Column Broser'.

I'm not too crazy about iCal 'improvements', but that went south when I switched from Tiger. I don't use Mail - Thunderbird instead. Mail kept hiding my e-mail someplace that I couldn't find.

Same thing with iPhoto - I use Adobe because it doesn't force me to use Apple's 'libraries' - when there is a perfectly good file system.

There's more, but that's more than enough carping for now.

Apple is getting closer to being the New Microsoft.

Missing Method in Python

2009-10-16T09:43:00.006-06:00

Preamble - optional

Ruby has a whole boat load of methods which allow a programmer to overload and override all kinds of stuff. One of them is 'method_missing', which is a method which is called if you invoke 'foo.bar()' and 'bar' is not the name of a method in 'foo'.

The Rails folks use it to transform function names like: find_foo_by_date_and_color(date, color) into a parameterized sql call, thus trading speed for apparent code clarity, added confusion for humans trying to understand the code, and adding inches to their stature as cool programmers.

You probably get from this that I think the Rails people overuse things like method_missing.

You're right.

But that doesn't mean it isn't useful.

So to add inches to my cool programmer stature, here's a case I think is justified - and how to do it in Python, which doesn't directly support method_missing (at least they don't seem to admit it - as far as I was able to find)

The Problem

I'm writing a little local HTTP server for my YASiteKit CMS/web-site kit thing (http://yasitekit.org) and decided to make it HTTPS capable. This turned out to be just a few lines of code in Python 2.6, but it broke the server. It turns out that while SSLSocket objects support some 'file-like' methods, they don't support all of them - such as readline() - which are used in the bowels of the HTTP server library.

The 'obvious solution' is to wrap the SSLSocket in something which adds the necessary methods.

But I'm lazy - in a sense - and decided to try to do this using a Pythonic equivalent to Ruby's missing method machinery.

Python has a rich set of special object methods for defining operations and the like, but it doesn't directly support a missing method call. It does support customizing attribute access so that you can define dynamic attributes - that is attributes which don't actually exist, but have computed values or that you want to create and delete dynamically after compile time.

I was stuck on how to pass arguments to an arbitrary method of a wrapped object instance until I realized that method invocation in Python is a two step process:

look up the attribute
call it with arguments

So the solution is easy:

wrap the object by passing it to the object constructor
create a __getattr__() method which returns getattr(wrapped-instance, method-name)
the Python interpreter then completes the call, with arguments, on the bound method.

If the method exists in wrapped-instance, then the returned value is the method, bound to the wrapped-instance. If not, then wrapped-instance throws an AttributeError exception.

It doesn't interfere with the wrapper's methods because __getattr__() is only called if the attribute is not found.

Besides avoiding writing a lot of method wrapping code, this has the advantage of telling me exactly which methods I have to implement - because when they are called on my wrapper, the wrapped object throws up.

Here's the sample code I wrote to verify the method:

class Floof(object):
var = 'This is Floof'
def __init__(self, var):
self.var = var
def func(self, var):
print('\nThis is Floof.func called on self')
print('Floof.var: ', Floof.var)
print('Floof.self.var: ', self.var)
print('Floof.func(var): ', var)

class Foo(object):
var = 'This is Foo'
def __init__(self, var, floof_instance):
self.var = var
self.floof_instance = floof_instance
def __getattr__(self, name):
print('\nFoo.__getattr__(%s) called' % name)
return getattr(self.floof_instance, name)
raise AttributeError('Foo instance: Attribute %s not found' % name)
def foo_func(self, var):
print('\nThis is Foo.func called on self')
print('Foo.var: ', Foo.var)
print('Foo.self.var: ', self.var)
print('Foo.foo_func(var): ', var)
def wrapped_func(self, var):
print('\nI\'m wrapping Floof.func()')
self.floof_instance.func(var)

floof = Floof('I\'m a Floof')
floof.func('calling floof.func() directly')
foo = Foo('I\'m a Foo', floof)
foo.foo_func('calling foo.foo_func() directly')
foo.func('calling floof.func() through Foo wrapper')
foo.wrapped_func('calling floof.func() via a wrapper method')
foo.flob('calling foo.flob() which does not exist')

Here's the output:

This is Floof.func called on self
('Floof.var: ', 'This is Floof')
('Floof.self.var: ', "I'm a Floof")
('Floof.func(var): ', 'calling floof.func() directly')

This is Foo.func called on self
('Foo.var: ', 'This is Foo')
('Foo.self.var: ', "I'm a Foo")
('Foo.foo_func(var): ', 'calling foo.foo_func() directly')

Foo.__getattr__(func) called

This is Floof.func called on self
('Floof.var: ', 'This is Floof')
('Floof.self.var: ', "I'm a Floof")
('Floof.func(var): ', 'calling floof.func() through Foo wrapper')

I'm wrapping Floof.func()

This is Floof.func called on self
('Floof.var: ', 'This is Floof')
('Floof.self.var: ', "I'm a Floof")
('Floof.func(var): ', 'calling floof.func() via a wrapper method')

Foo.__getattr__(flob) called

AttributeError: 'Floof' object has no attribute 'flob'

function __getattr__ in missingmethod.py at line 52
return getattr(self.floof_instance, name)

See - It works!

A Story

2009-07-14T09:53:00.005-06:00

Herman

How it Started

The day the Lemmings ran over the cliff, Herman forgot and went to the park.

When it got dark, he went home – but there was nobody there. That’s when he remembered about the Big Cliff Run.

They were all dead now.

That’s the way it is with Lemmings. He remembered.

Then he cried.

The next morning, when he got up, Herman remembered.

He couldn’t understand what had happened. All his life he’d been taught how to be a good Lemming. Maybe he wasn’t a Lemming?

He felt himself. Felt like a Lemming.

He looked in the mirror. Looked like a Lemming.

He was hungry – just like always – and just like a Lemming.

Then how come he’d forgotten the Big Cliff Run? Wasn’t that supposed to be Instinct? Wasn’t it built into his Jeans? (or was it Genes?) Wasn’t it Inescapable and the Source of All Sorrow?

How could he just forget something that was as much a part of him as his nose?

Something was wrong.

So Herman set of to see Marvin – the big, bulky Lemming On the Hill. Marvin knew everything. Marvin knew all the Old Stories from the Old Times by the Old Lemmings who Knew Everything. Marvin was a Great and Good Lemming. The Greatest and Goodest.

Marvin was so Good that he didn’t go on the Big Cliff Run. It made him ever so miserable not to Go – to hold himself back – but he did it for the Good of Every Lemming!

He had to. He couldn’t let Lemmingness be Lost! Someone had to sacrifice for the Good of All Lemmings. Someone had to tell the Stories. Someone had to make sure All the Lemmings made the Big Cliff Run.

It was a Heavy Burden, but Marvin kept the Faith and Did His Duty.

Marvin

Marvin wasn’t happy to see Herman. In fact he frowned and said something Herman hadn’t ever heard before (which was good, because then Herman didn’t understand what it meant and didn’t know how angry Marvin was).

“What happened,” Marvin asked.

Herman blushed under his fur. “I just forgot,” he said.

”!!!!!,” Marvin exploded!!!!

Herman stuttered, “I just woke up and it was so nice and sunny and so I though ‘I’ll go to the park’ and I didn’t think anything else and so I went and it was nice and the birds were out and singing and the grass was soft and . . .” It all came out in a rush.

Marvin glowered.

“Don’t you remember your lessons?” he asked. “Didn’t I tell you to repeat the Lemming Mantra every morning and every night?” “Didn’t I”

“Yes,” said Herman.

“Well? . . .”

“I just forgot. It was such a nice day, . . . and I just forgot,” Herman said lamely.

“Hmmph!!!” said Marvin.

As Herman trudged down the hill he saw Lonny, Marvin’s apprentice and student dragging a big bag of Cheetos up the hill. Marvin was too fat (oops! too important) to go get his own food, so everyone brought him what he needed. Lonny and Marvin were great Lemmings to hold themselves back and stay alive year after year, thought Herman.

Sighing, Herman walked slowly home.

“I’ll remember Next Year,” he though as he sadly trudged along.

Interlude

Over the next year, the few surviving Lemmings rebuilt. They had new babies who quickly grew up and had more. Marvin and Lonny were busy teaching and preaching. Herman attended all the lessons. He worked very hard at fixing himself so he’d be a Good Lemming. He repeated the stories over and over again. He repeated the Lemming Mantra extra times every day.

But Herman loved going to the park and watching the clouds. Every day it was different. Every minute even.

He marked Big Cliff Day on his calender and as it approached, he worked extra hard at thinking Lemming Thoughts. He Felt the Sorrow. He Felt the Doom. He was a Lemming. He knew what had happened last year – he just hadn’t taken it seriously enough. Hadn’t filled his mind with Lemming Thoughts. He hadn’t repeated Over and Over again what a Lemming did and how a Lemming Thought and how Inevitable everything was.

Lemming’s can’t Change! It’s how they’re Built. It’s Innate. It’s Inevitable. It’s who we Are. It’s Lemmingness!!!!

The night before the Big Cliff Run, Herman worked extra hard at remembering. He laid out his special shoes and everything. Then he went to bed.

The next morning was a beautiful day. Herman woke up earlier than usual. Earlier than everybody else, in fact. Since he was early, he decided to go to the Park One Last Time. So he did.

It was so nice. The clouds were making beautiful swirls. It was almost hypnotic. Lovely. Beautiful. Peaceful. Happy.

Herman woke up suddenly.

He remembered – that he’d forgotten again.

He went home.

Everyone was gone and he was alone again.

This time, Herman sat down and shook his head.

“Something isn’t right,” he thought. He started to go to Marvin, but then stopped. He had an idea – well maybe just the start of one.

Could Marvin be wrong? No!

The Idea

Herman thought and thought and thought and thought and then fell asleep. Thinking is tough work and he wasn’t used to it. All his life Herman hadn’t actually ‘thought’. He’d been a Good Lemming and he’d only repeated what he was told. He’d been told Over and Over what to repeat, so he did it. He repeated the Lemming Mantra. He repeated the Song of Sorrows. He repeated the Five Reasons for Melancholy. He kept the Record of His Life in a little book – where he wrote down Every Bad Thing. He read the book from cover to cover every week (it took longer and longer the more he wrote into it – in fact it was getting to where he didn’t have time to do much else)

Herman did his duty and met with the Other Lemmings. They all took turns telling about the Bad Things, reminding each other how terrible life was. Over and over. And when one of them saw something nice and beautiful, they made doubly sure that everyone took a turn telling what was bad about the Nice Thing and how disappointing it would be. They were very, very good at protecting each other against the Unbearable Pain of False Hope.

But as the year wore on, Herman noticed how much work it was to remember to feel bad. He was tired all the time. He didn’t go to the Park as much.

After a while, he found himself just sitting in his chair looking out the window. Not repeating the Mantra. Not reading the Bad Things book. Just sitting and looking.

That’s when he started really thinking.

What if . . . what if . . . what If . . . WHAT IF LIFE REALLY WASN’T SO BAD?

He used to like going to the park. It was fun. It was fun because he wasn’t Thinking the Bad Thoughts. He wasn’t Remembering the Sorrow.

He had the Thought!

“Maybe thinking the Sorrow Makes the Sorrow.”

And then the Discovery.

“Maybe if I don’t think it all the time, it will Go Away”

Interlude (again)

Herman didn’t know what to do. He’d never had a thought on his own before. He worried and worried that it was Wrong. Marvin Thought and Marvin Taught. Lonny will be the new Thinker and Teacher – once he’s ready. “I’m not Smart enough,” thought Herman.

But the Thought wouldn’t go away. It kept coming back.

It was strange because Herman didn’t have to work at thinking this thought. It just kept coming back. Not always the same and not very clear. It was more like a feeling that said “It doesn’t have to be this way” and “That’s a nice thing, let’s think about it for a while”.

It wasn’t so much the Things he was Thinking about, but how the Thoughts felt. Like the nice fluffy clouds and the warm sun. Nice and peaceful. Calm and serene. Not at all strong and hurting – like he was supposed to think.

Herman finally decided he needed to see Marvin.

So he went.

Marvin was even more displeased. He barked and preached at Herman for over two whole hours. He wouldn’t answer any of Herman’s questions - in fact he just ignored them. Then he told Herman to Go Home, Believe, and Behave.

As he trudged home, he say Lonny dragging another bag of Cheetos up the hill to Marvin. Lonny was tired, so he was happy to stop and talk. Herman told him everything he’d tried to tell Marvin. He finished up with “I just don’t understand. I just don’t understand. I just ,” at which point Lonny interrupted him.

“Look, Herman,” he said, “I like you, even though you’ve got some screwy ideas and I wish I could help you. Marvin’s getting kind of old and he just doesn’t have the patience he used to have. But I don’t have the time right now – but if you’d like to borrow my Book, I’ll let you have it for a bit. I’ve got to have it back on Tuesday, but maybe it will help.”

A book? Herman had never heard of the Book before. He knew how to read and write – like all good Lemmings, but he didn’t know there was a Book. He thought Marvin had learned the Stories from his Teacher and his Teacher from his Teacher and like that.

Lonny was holding out the Book, so Herman took it, mumbling something to cover his embarrassment, and scuttled home.

The Book

When Herman got home, he opened the Book and started to read. It made his head hurt because it didn’t make sense.

“I’ll never understand this by Tuesday,” he thought, so he decided to make his own copy. He got to work, copying the Book.

After that, Herman spent every day reading his copy of the Book. He kept looking for what Marvin had Taught him, but it never really seemed to fit.

He read “Our Thoughts make our World, so Choose your Thoughts Carefully” and he remembered how Marvin had droned over and over what thoughts to think. But it just didn’t feel right because the book said, “The Thoughts you hold Dear will Be your Life – so Think Thoughts of Peace and Joy”.

He read and read and thought and thought and became more and more confused. He thought of going to the Great Marvin, but that frightened him. Then he thought of Lonny. Maybe he could explain it.

So, the next day he waited for Lonny to come by on his daily quest for Cheetos for the Great Marvin.

“Excuse me Lonny, I have some questions about the Book”

Lonny stopped and looked puzzled – or maybe was shocked and stopped so he wouldn’t fall down.

“What Book?” he asked.

“The one you lent me and I copied,” Herman replied.

“The one you copied?” asked Lonny. “That’s impossible. There is only one Book. There can’t be two – it’s Forbidden.”

Startled, Herman looked confused. “But I had to copy it so I could get it back to you on Tuesday. I couldn’t understand it that fast.”

Lonny looked worried. “Please don’t tell anyone.” (meaning Marvin) “Please don’t, tell me you won’t.”

“Sure,” said Herman. “But will you help me?”

“With what?”

“Well, I know what I was taught and I know what the Book says and they’re not the same,” Herman began.

“Oh yes they Are!!!” Lonny said. “You just don’t Understand. That’s why there’s only One Book. You have to Taught How to Read It.”

“Will you teach me?” Herman asked.

“I can’t. You’re not the Student – I am,” Lonny said. “I can’t Teach until I’m the Master and the I can Only Teach the Student. That’s how it is.”

“Well, the Book says that the Master taught his students,” Herman began.

“That’s Right,” said Lonny, “and that’s why there can only be One Student and No One can Teach except the Master.”

“But, it says in the Book that the Master taught lots of students - not all of them learned well, of course, but he taught everyone. I don’t understand.”

“That’s right, you don’t. You see, the Master could only really Teach the Students who learned, not the other students. And those other students went around teaching others Wrong Things.”

Herman scratched behind his left ear. “Hmm. So now the Master doesn’t let anybody else Teach or read the Book because they might get it Wrong?”

“That’s Right,” Lonny replied.

“And Marvin knows How to Read the Book because . . .” Herman trailed off, confused.

Lonny sighed and sat down. “Look,” he said, “Marvin’s Master Taught him and he had a Master who Taught Him and so on – all the way back to the Master who Wrote the Book. That’s how Marvin knows how to Read it and How I’m Learning too.” Lonny looked compassionately on Herman. “You’ll just never understand.”

“Will you help me?”

“I’m sorry, but I can’t,” Lonny answered.

“Well, how about just one question then,” Herman asked.

“OK. Sure. But, just one.”

“Marvin teaches us to remember the Bad things, to Write them Down in our book of bad things and to read it every week. And anytime someone thinks that Bad won’t happen, it’s our Duty to remind them of All the Bad things. The Book says that what we Think is our World. Doesn’t that make our World Bad instead of Good?”

Lonny sighed. “See, you just don’t understand. What’s the worst thing that can happen to you?”

“I really don’t know,” Herman said.

“Think about this: suppose you thought you would get a great big Strawberry Shortcake for dinner tomorrow. You’d spend the whole day and night just thinking about how great that would be. You’d plan the whole next day how good it would taste and comfy and fat your belly would be. Can you imagine that?”

“Oh boy, yeah. I can almost taste it now.” Herman started to drool.

“Ok,” Lonny continued, “now think about how disappointed you’ll be when you don’t get it or if it’s filled with ants or the Strawberries are rotten. Think about that.”

“Yech! That’s terrible,” Herman responded. His fur actually started to wilt.

“So you see,” Lonny lectured, “it’s much, much better to never get your hopes up because that makes it so much farther to fall down. You see thinking Bad Thoughts protects use from disappointments like that – and even worse.”

“That’s why thinking Bad thoughts is Really Good,” Lonny finished. “I hope that helps. Do you see why someone has to Teach you How to Read the Book? And why it’s not Good for You?”

Herman just mumbled.

“Well, I hope that helps you – but I have to go about my duties,” Lonny said, standing up and starting to walk away. Then he stopped, “and remember, please don’t tell any anybody.”

Herman started. “Oh, sorry. Yeah, thanks. No, I won’t tell anyone. Bye”

So, Lonny went off to do his important tasks and Herman sat and sat some more. And thought and thought some more. What Lonny said made some sense, but something just didn’t seem right.

Herman went on like this for a long time. Reading the Book and thinking about what he had been Taught and what Lonny had said. His fur got dull.

One night, very very late, Herman snapped. “I’ll never make sense of it,” he said out loud.

From that night on, Herman quit trying to be a good Lemming. He stopped doing what he’d been Taught. He forgot about the Book. He’d decided to just live the best he could.

Herman no longer wrote in the book of Bad Things. He no longer recited the Lemming Mantra. He began to forget the teachings. He didn’t read the Book – in fact, he forgot about it.

Each day Herman went to the Park. He looked at the trees and the grass. Slowly, the Bad Thoughts didn’t come so often. They didn’t Stay so Long. He didn’t feel so bad. The Great Sorrow lifted.

The Realization

One day Herman saw the Book. Wrinkling his brow, he picked it up and opened to a page at random. It made sense. It was simple. Why hadn’t he understood it before? You really are what you Think. And you can Chose what you Think.

Lonny was Wrong. Marvin was Wrong. Thinking Bad Thoughts didn’t Protect you, it made a Bad World. If the Strawberries are rotten, then just forget about it: there will be fresh ones some day.

The Answer is to Forget about the Bad and Think about the Good!

Herman couldn’t wait to tell everyone.

He ran outside.

The End

Well, I won’t tell about how everyone laughed at him. How no one believed. How Marvin was even More than Displeased (luckily he was too fat to move or he might have squashed Herman flat). Or Lonny’s look of pity.

Nobody believed him. After all, the were Lemmings and he was only a lemming – not a Real Lemming At All. He didn’t understand the Burden that came with Lemmingness – the Great Sorrow – the Inevitability of the Great and Senseless Death, crushed at the bottom of the Big Cliff. He didn’t understand.

On Big Cliff Day, Herman walked out to the side of the road and shouted to them as they ran by: “You don’t have to go! Just go to the Park! Just Forget the Bad Stuff! You have a Choice!”.

The ones who heard him clapped their front paws over their ears so they couldn’t hear – which made it very hard to run with their noses in the dirt. But they didn’t care. They wrapped their misery around themselves proudly and – bleeding and with broken teeth – plowed on toward the cliff.

All the while, Marvin sat on his hill – praising their Lemmingness, while he ate his (final) bag of Cheetos. He would not live to see another Big Cliff Day. It was now Lonny’s turn to be the Master.

It slowly dawned on Herman that they would defend their misery with their last dying breath rather than change their minds. “They think they are Lemmings,” he said, softly, under his breath, “They think that if they change their minds about something, they won’t be themselves.”

So, Herman walked to the Park and watched the day go by.

He was sad.

Then he realized that, while he now knew, he couldn’t help them. They didn’t want a better life. They wanted to feel secure in the life they already had – so they only listen to Lemmings who will tell them that they are right. Lemmings who confirm their misery. He was just a lemming.

But that’s OK.

So, he decided – right there – that if anybody asked, he would tell them, but that he wouldn’t bother anybody any more. (the thought of those Lemmings pushing their noses in the dirt was just too much to bear, even though it was kind of funny)

The End

Why XHTML is Wrong and HTML 5 is Right

2009-07-14T09:30:00.005-06:00

I'll admit it: I fell for the XHTML hype and I've tried to convert myself to writing XHTML rather than HTML. The argument sounded good, but . . . I was wrong.

Then about a month ago I read almost all of the HTML 4.01 spec - not the digestions that you find in books like Everything You Ever Wanted to Know to Be an HTML Expert!!!!! in 5 minutes a Day!!!. I read the real spec. I actually spent about 1/4 of my time reading the DTD.

[I thought I needed to write an HTML parser in PHP - but that was just another programming hallucination which I solved in a much simpler, but vastly more limited and restrictive way]

Anyway, while doing that I realized that writing an HTML parser is Hard. Whereas writing an XML parser is EASY.

And that - in a succinct nutshell - is the only argument in favor of XHTML.

And, along with everybody else, I blindly followed that Pundant-centric reasoning.

But lets think about it: How many times do you Write an HTML Parser? For most of us, it's Zero.

OK, how many times do you have to read and understand HTML? For most people it's still Zero, but for a Lot of Us it's 'All the f****ing Time!'.

So - should (X?)HTML be Human Parse-able or Machine Parse-able?

That is: Should (X?)HTML be Easy for Us to Read or Easy to Write simple Parsing Programs for?

If you don't know the answer to that one, ... I was about to write something insulting, but that's not fair. Let's look at some history:

1. RPN [that's Reverse Polish Notation]. It's easy to parse, but hard to read. It has a long history, but even the original HP calculator [showing my age] which was a commercially successful RPN calculator had to succumb and convert to algebraic notation. (and as far as I know, the only commercially successful one)

2. The FORTH family of languages. Apparently this never really dies. FORTH was a stack based language which is really easy to write a parser for, but almost impossible to understand. Why? All the operations either push something on the stack or pop some stuff off, transform it and then push it back. Us Humans are NOT Stack Machines, so we can't track the state of the Stack.

3. Postscript - Same problem as FORTH, but it's successful as an intermediate language to describe a rendered document. Nobody actually programs in Postscript - we all use tools which output to Postscript and then feed that to a Postscript interpreter. [anybody every try to read that stuff?]

4. Etc

Point Made?

But, you say, HTML Sucks!!!

Right!!!

Lets get behind HTML 5 and Fix it. XHTML Sucks too.

PHP, Language Design, and Confusion

2009-05-01T14:05:00.010-06:00

References

When I wrote some iterator code which returns a reference to an internal array in an object I discovered an interestingly easy to introduce bug. Without getting into the details, I left out a '&' and clobbered my underlying data structure.

Here's the offending loop:


for ($row=&$obj->firstRow();$row;$row=$obj->nextRow() {
  other stuff
}

where firstRow() and nextRow() are iterator methods defined on the object which do exactly what they sound like. [oh yeah, they both are defined as returning references]

Spot the bug?

I left off the '&' between the '=' and the '$' in the update part of the for loop. This causes $obj->nextRow() to overwrite the firstRow.

Strangely enough, if I wrote this in C I wouldn't have made the error. The equivalent C code would be:


for (ptr=*obj->firstRow();ptr;*ptr=*obj->nextRow() {
  *ptr-> whatever
}

Notice that I have to explicitly dereference the pointer ptr in order to clobber the initial element.

So this covers PHP and the Confusion.

Here's where the Language Design comes in:

C makes an syntactic distinction between accessing a pointer and dereferencing it.

PHP doesn't. If a variable contains a reference then assignment deferences it implicitly and silently.

Consequently, you can't really understand a chunk of PHP without reading all the definitions of the variables which preceed the chunk you're looking at. This is a bad thing and it violates that tried, true and mostly forgotten maxim of 'code locality'.

This also leads to a lot of bugs - and 'Bogus' bug reports - involving references. [just check out bugs.php.net - 655 Bogus bugs relating to 'reference']

I think PHP would be a lot easier to understand if there was a syntactic difference between assigning to a variable and assigning to the referrant of a variable.

Practically speaking, I don't think it will happen because of 'backward compatibility' and 'NIH'.

What I'd like to see is a dereference prefix which is accepted in PHP 5.x and becomes mandatory in PHP 6. There should also be a warning option which generates a warning if a variable containing a reference is assigned value and the variable is not prefixed.

Here's my first (and only) choice:

&$foo

This is currently used to denote a reference when used on the right side of an assignment. It should be a simple matter to extend this to dereferencing on the left side and in expressions.

Efficient Code - PHP

2009-04-23T22:32:00.004-06:00

I guess I'm a nut about execution speed. Well, not if it gets in the way of clarity, but I hate writing slow code.

But . . . always a 'But' . . . it's hard to know what is slow and what isn't. Seems like it should be easy, but I never know until I measure.

Here's an example:

The PHP manual page for preg_split() says that you shouldn't use it unless you need the flexibility of regular expressions. You should use explode() or str_split(), because they are simpler and therefore faster. I believed that.

Then I thought I'd like to see how much faster, so I wrote a couple of loops which split a comma separated string with both preg_split() and explode() to see how much I was really losing.

Here's are the average times of 10 passes through 1,000,000 splits using each:
preg_split() - 3.51239209175 seconds
explode() - 4.33661820889 seconds

preg_split() is about 23% Faster than explode().

I'm still not sure I believe it. Here's the code - so you can try it yourself.

And, please let me know if you see anything I did wrong.


$str = 'this, is , a , string, with , commas, in ,it';
$count = 1000000;
$passes = 10;
$times = array(xdebug_time_index());
$avg = array('preg' => array(), 'explode' => array(), 'map-explode' => array());
function dt($offset = 1)
{
  global $times;
  $last = count($times) - 1;
  return $times[$last] - $times[$last - $offset];
} // end of dt()

for ($pass=0;$pass<$passes;$pass++) {
  echo "Pass $pass\n";
  $times[] = xdebug_time_index();
  for ($i=0;$i<$count;$i++) {
    $v = preg_split('/^\s*,\s*/', $str);
    unset($v);
  }
  $times[] = xdebug_time_index();
  $avg['preg'][] = dt();
  echo "$count preg_split's(regx, str): " . dt() . "\n";
  
  $times[] = xdebug_time_index();
  for ($i=0;$i<$count;$i++) {
    $v = array_map('trim', explode(',', $str));
    unset($v);
  }
  $times[] = xdebug_time_index();
  $avg['map-explode'][] = dt();
  echo "$count array_map(trim, explode(',', str)): " . dt() ."\n";

  $times[] = xdebug_time_index();
  for ($i=0;$i<$count;$i++) {
    $v = explode(',', $str);
    unset($v);
  }
  $times[] = xdebug_time_index();
  $avg['explode'][] = dt();
  echo "$count explode(',', str): " . dt() ."\n";
}

echo "Averges\n";
foreach ($avg as $key => $ar) {
  echo "$key average of $passes trials of $count splits: " . array_reduce($ar, create_function('$a,$b', 'return $a+$b;'), 0) / count($ar) . "\n";
}

Documentation - It's Part of the Design Process

2009-04-21T22:25:00.008-06:00

I was documenting some code I just wrote to create a packing list for an internet store we'll be opening soon. It's kind of a hairy problem to come up with a semi-optimal allocation of goods into boxes - but absolutely necessary to do right if you want to accurately estimate shipping and keep costs down.

It took me about a week to really get my head around the problem so that I could solve it. The actual solution now looks pretty obvious - but it won't in six months or a year.

So I document all my code - profusely, but compactly.

A Strange Thing happened while I was writing the doc: I found and corrected a whole bunch of bugs - both coding bugs, 'bugs of omission', and design bugs. That's when I realized that I use the very act of documenting code as part of my design iterations.

So, I started thinking about what I do and why I do it the way I do - which is (as usual) a bit contrary to 'accepted best-practice'.

First of all, there are basically Three Types of documentation in code:

The Code Itself
Comments
Doc

The Code: My code is semi-self documenting. I do the usual stuff: identifiers are semantically related to their use [function save_data() - etc]; use of white space; consistent breaking of long lines and breaks at binary operators; consistent indentation.

But you can do more.

Simple, direct, and consistent code does wonders for readability. Consistency in within-loop logic [use if elseif elseif ... or switch case case default or if () { stuff; continue;} if () { stuff; continue;}]. I don't think it matters what style you use as much as consistency and simplicity.

Small is usually good as well. We used to talk about 'locality of code' meaning that our code was written in chunks which weren't 'too long.' 'Too long' usually really meant 'it fits in the window of my editor' so that you can stare at it and understand the whole thing. There's also 'too short', but the Ruby on Rails guys have the market on that with their obsessive DRYness.

Anyway, the Code itself is always the final authority on what the program does - so it's the most authoritative documentation.

Do yourself a favor and make it as readable as possible - you might have to fix it later.

Comments: Helpful hints which make the Code more readable. Some (IMHO) IDIOT came up with the idea that Comments Must be Set Off with Big Boxes of Stars or something like that.

Why?

Well, I'll tell you.

We used to write programs on Paper and then punch them into Punch Cards and run them through a Card Reader and then get a Big Wide Printout on Greenbar paper and do our debugging at a desk. We'd page through this Paper with a pencil and write revisions to our programs. The paper was 14 inches wide [132 columns] and 11 inches long [60 lines at 6 lines per inch with 1/2 inch top and bottom margins].

In those days it made sense to Mark Off the comment blocks. It made them easy to spot - both for someone reading the code and for the Boss as he walked by your desk. [You'd get demerits for not commenting your code, so it was a good idea to make it easy for him to see].

Things have changed. You can easily flip around in an on-screen editor, but you can't very well write on it. It works a lot better if the information in the editor window is very dense. The way to do that is to write meaningful comments with as little excess white space as possible.

Now I use blank lines to separate logically disjoint chunks of code and preceed them with comments - if appropriate - but I don't waste a lot of screen space blocking off comments.

I think we need a very high Signal to Noise ratio - and big blocks of comment markers are just noise.

I don't use PHPDoc because those guys seem to think that the Documentation and the Comments are More Important than the Code itself. I don't. I want as much Code on the screen as I can get if it is blocked out so that the logic is obvious and commented as needed.

The Doc: The Doc is narative description of what the code does, how it does it, how to use it, calling parameters, etc etc. Can't live without it.

I've experimented with all kinds of methods. I used to write UNIX style man pages - and they are still a really good format. But now days, everything has to be in HTML, so I've changed.

There are a lot of Documentation Project things - like PHPDoc - but I don't use them.

I think they're too complex and they are usually specific to one language. Besides, they just aren't necessary for good, clear documentation.

Like many people, I work in several languages [one reason I'm bald] so I need a multi-lingual documentation system. I also want something which doesn't take up a lot of space in my program files and which is really simple. Here are my 'Must Haves':

Must allow the Doc to be imbeded in the same file with the code
Must produce HTML
Must be flexible
Must be very Simple

I couldn't find anything which didn't take multiple days [or weeks] to learn, so I wrote up something to pull Textile makeup out of a simple text file and that's what I use.

Frankly, I'm smart enough and write well enough to describe what some code does. So are you. Just pretend you're writing for yourself about a year from now when you have to find a bug and fix it in record time - or you'll be fired.

What kind of Doc do you want then? Clear, correct, self-contained without a lot of references jumping around, etc etc. Write something which will save your butt when the crunch comes and you're fine.

Design?

So what does this have to do with Design?

As I write my Doc - part 3 of the Code Documentation - I find myself thinking about how the whole thing works and how understandable it is. It can get kind of contemplative.

Another thing which happens is that I find myself wondering why in the hell I wrote something and what it really does - so I go through the logic again.

There's a different perspective and attitude when reading code to understand and clearly document it than there is when writing it initially. At least there should be.

When I write code, I get involved in the microscopic detail, in solving the problem, in creating the logic and the loops.

When I read code to document it, I get involved in the overall strategy of the solution and the clarity of the logic. It's just different.

Anyway, that's how I think about Documenting code now days.

Oh, if you're interested in my document extractor - I'm planning on dumping on the download page of http://www.clove.com. It's written in Python, so it should run about anyplace.

How can you screw up XML? Part II

2009-03-03T09:22:00.004-07:00

XML is overly wordy. It has lots of required text.

I think the theory is - if there really is one - that longer words make the computer-code more easily understood. That was certainly true when we were using FORTRAN with 6 character variable names and Basic with 2 characters - but there's a limit. That limit is probably around 32 characters - 1/3 of a screen line - at which point you need some white space and punctuation.

XML doesn't provide for white space, just punctuation. As (semi-)intelligent creatures, we need white space to clump symbols into recognizable things. It seems to be how we parse. Computers don't distinguish, so . . . They can read XML, but we can't. [maybe it's really a plot by the machines and W3 are a bunch of Cylons?]

So maybe the extra stuff is supposed to make XML more reliable? To get there, we have to talk about two slightly different subjects: Bandwidth and Information Theory.

Bandwidth measures how fast we can transmit messages. The bigger the bandwidth, the faster the electronic wiggles are and so the more Bits we can represent in a given chunk of time - say a second. Big Bandwidth is Good.

Effective Bandwidth is the fraction of the real Bandwidth you get to use for your stuff - the content you want to see, transmit, or use - read streaming video from hulu.com. The more non-Content characters in the Protocol [read XML] used to encode your 'stuff', the lower your Effective Bandwidth. [You pay for Real Bandwidth, but you Get Effective Bandwidth - it's kind of like sales tax or Net Income after Income Tax]

Information Theory studies how to transmit Information in the presence of Noise. It turns out that you can always get your message across accurately - most of the time - if you use a fancy enough code. A Code takes a simple message and adds a lot of extra bits which allow the receiver to tell if the message was messed up [received a 0 where there should have been a 1] and to reconstruct it. When you have more noise, you have to add more reconstruction bits. That makes the message longer. So Information Theory says - if you want reliable communication, you have to allocate some of your Bandwidth to these reconstruction bits - called Redundancy - so that your Effective Bandwidth is lower than your actual Bandwidth.

Now let's apply this to XML.

XML adds extra stuff to create a rigid structure for you message. This has Nothing to do with Information Theory because XML is transferred over TCP - which is a Lossless Protocol - meaning that, if the message gets there At All, it's guaranteed to be OK. There's No Noise on TCP. [The TCP protocol has already eaten up the Bandwidth required by Information Theory to get your 'stuff' to you]

The XML extra stuff is there so computer programs can easily parse the message and use it's Content once it gets there.

XML's 'extra stuff' shares the Same Bandwidth as the Content inside the XML message.

I ask you: what's more important: the Content or the 'extra stuff'?

Personally, I think the 'extra stuff' should be as small and efficient as possible so we can use as much of the Bandwidth for Content.

W3 must think that the 'extra stuff' is more important than content because they make their protocols as bulky as they can.

Don't believe me? go to www.w3.org and read some of their specs - try name spaces or the RDF spec or just about anything. All Structure with minimal space for Content.

Why do we put up with this?

How can you screw up XML?

2009-03-03T08:13:00.010-07:00

XML is about the simplest thing around. How can you screw it up?

I'm writing some metadata-extraction-from-image code and started looking at Adobe's XMP.

XMP is written using the W3's RDF spec - [almost wrote RDF framework, but that would then be Resource Description Framework framework (RDF²), which might not be the same thing].

RDF defines a 'Framework' for writing machine parseable statements of the form:
Tim has a bike.
RDF calls:

Tim the subject

has the predicate

bike the object

It makes you write URI's for both the subject and predicate - and have translation syntax between real words and the URI's.

Here's an example from the RDF Primer. It says: 'http://www.example.org was created on August 16, 1999'

<?xml version="1.0"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
           xmlns:exterms="http://www.example.org/terms/">
 <rdf:Description rdf:about="http://www.example.org/index.html">
     <exterms:creation-date>August 16, 1999</exterms:creation-date>
 </rdf:Description>
</rdf:RDF>

The RDF is 5.9 times LONGER than the English Language Sentence. That's an increase
in text of about 83%. Or, to put it another way, a BANDWIDTH UTILIZATION of about 17%.

For What Gain?

Nothing. And it takes them 6 LONG RFC style documents to define this messs. That's 6 Long, Boring documents with much repetition and pedantic phrasing with many MUST's and SHALL's and MAY's.

But it can be parsed by a machine - if you can understand the spec well enough to write the code.

Why take something so simple and make it so incomprehensibly complex?

But - Believe it Not - I digress.

XMP is written using RDF [why re-invent the wheel when you can use somebody else's debacle and make it worse].

I'm not going to get into XMP - but at first glance it looks like they're using attributes for data, XML entities for data, and RDF nested structures for data - with NO obvious logic as to when and where these choices are made. To further mess it up, everything uses XPATH name spaces - which my XML parser translates back to URI's [which point to nothing, but are long and look cool], like a 'good parser should' - which obfuscates the already obfuscated and bulks out the fluff to content ratio admirably.

Yech!!!!!!!

Here's how I think XML intended to encapsulate a website's creation date:

<site-info site_name="www.example.com">
<creation-date>August 16, 1999</creation-date>
</site-info>

It's machine parse-able. It's (almost) human readable. It only wastes 50% of the bandwidth - as opposed to 85% using RDF.

How about JSON - where the entire spec fits on one web page:

{
 "site_info": {
   "site": "www.example.org",
   "creation_date": "August 16, 1999"
 }
}

PHP Input Cleaning

2009-02-17T10:17:00.009-07:00

I've been buried in my be-all, end-all, does-everything CMS / e-commerce site.
I thought I'd come up for err (or air) and here's what happened.

Below is an object which does lazy input cleaning of GET and POST data.

You use it by creating a new object [surprise].

You access cleaned data as attributes of an object instance.

All the cleaned data is cached in a class variable, so you can either pass around
a global or create a bunch of instances and everything will work - within the same
instance of the PHP program, of course. Once the Object
is required, it acts kind of like a Singleton Pattern.

You can restrict the query sources to GET, POST or REQUEST and control how it
reacts to undefined query parameters.

The documentation is in a comment at the top in Textile.

In some ways, I think this is kind of neat, but on the other hand it's
really depressing how much effort goes into something with such a simple
objective. Seems like there should be an easier way.

Cheers,


<?php
/*
#doc-start
h1.  RequestCleaner

Created by  on 2009-02-14.
Copyright (c) 2009 Clove Technologies, Inc. All rights reserved.

h3. Usage

Create a new request cleaner:

$rc = new RequestCleaner(sources, use_modes, error_mode) - where:

* sources is a comma separated string OR an array with one or more of: GET, POST, REQUEST
* use_modes is the bitwise OR of
** RequestCleaner::USE_HTMLENTITIES - process each parameter with _htmlentities()_
** RequestCleaner::USE_HTMLSPECIALCHARS - process each parameter with _htmlspecialchars()_
** RequestCleaner::USE_NL2BR - process each parameter with _nl2br()_
* error_mode - determines how the beast responds to errors - such as non-existing query parameters.
Use one of:
** RequestCleaner::RETURN_NULL - returns NULL
** RequestCleaner::RETURN_FALSE - returns FALSE - can be distinguished from NULL by '===' and '!=='
** RequestCleaner::THROW_EXCEPTION - throws an exception

For example:

$rc = new RequestCleaner('POST', RequestCleaner::USE_HTMLENTITIES | RequestCleaner::USE_NL2BR,
  RequestCleaner::THROW_EXCEPTION);


To get a 'cleaned' query parameter, use '$rc->parm', where _parm_ is the parameter name.
For example, if 'foo' is a POST parameter, then '$rc->foo' will return the value of 'foo'
from after running the 'cleaning' routines on it.

h3. Fine points

All Cleaned data is cached in a RequestCleaner Class Variable, as are the sources, use and error
modes. This means that a top level PHP program can set up the method of cleaning and
allowed sources and all included code which uses _any_ instance of a RequestCleaner
will use the same methods and cache.

Consequently, you can either create a RequestCleaner instance at top level OR in any included or
required file which needs to access query parameters. It doesn't matter.

Attempts to access undefined attributes generate an error - as specified by _error_mode_.

Query parameters which return arrays - as in <input type=... name="foo[]" ...> -
are turned into arrays of cleaned strings which can be processed using normal loops
and array_...() functions

#end-doc
*/

/**
* RequestCleaner(sources = NULL) - where sources defines a comma separated list of Super Globals
* for attribute values. Legal names are: POST, GET, and REQUEST
*/
class RequestCleaner
{
  const ERROR_MODE_MASK = 3;
  const RETURN_NULL = 1;
  const RETURN_FALSE = 2;
  const THROW_EXCEPTION = 3;
  const NORMAL = RequestCleaner::RETURN_NULL;

  const CLEANER_MODE_MASK = 0x07; // IMPORTANT: Change This as you ADD USE_modes
  const USE_HTMLENTITIES = 1;
  const USE_HTMLSPECIALCHARS = 2;
  const USE_NL2BR = 4;
  private static $use_modes = array(
    RequestCleaner::USE_HTMLENTITIES => 'htmlentities()',
    RequestCleaner::USE_HTMLSPECIALCHARS => 'htmlspecialchars()',
    RequestCleaner::USE_NL2BR => 'nl2br()',
    );
  private static $error_modes = array(
    RequestCleaner::RETURN_NULL => 'return NULL',
    RequestCleaner::RETURN_FALSE => 'return FALSE',
    RequestCleaner::THROW_EXCEPTION => 'throw Exception'
    );
  private static $sources = NULL;
  private static $source_names = array();
  private static $error_mode = NULL;
  private static $use_mode = NULL;
  private static $cache = array();
  function __construct($sources = array('GET', 'POST'), $use_mode = RequestCleaner::USE_HTMLENTITIES,
    $error_mode = RequestCleaner::RETURN_NULL)
  {
    if (!RequestCleaner::$sources) {
      if ($sources) {
        RequestCleaner::$sources = array();
        if (is_string($sources)) {
          $sources = preg_split("/,\s*/", trim($sources));
        }
        foreach ($sources as $src) {
          RequestCleaner::$source_names[] = $src;
          switch ($src) {
            case 'POST':
              RequestCleaner::$sources[] = $_POST;
              break;
            case 'GET':
              RequestCleaner::$sources[] = $_GET;
              break;
            case 'REQUEST':
              RequestCleaner::$sources[] = $_REQUEST;
              break;
            default:
            throw new Exception("RequestCleaner::__construct($sources): Illegal Source: $tmp");
          }
        }
      } else {
        RequestCleaner::$sources = array($_POST, $_GET);
      }
      if (RequestCleaner::CLEANER_MODE_MASK & $use_mode) {
        RequestCleaner::$use_mode = RequestCleaner::CLEANER_MODE_MASK & $use_mode;
      }
      if (RequestCleaner::ERROR_MODE_MASK & $error_mode) {
        RequestCleaner::$error_mode = RequestCleaner::ERROR_MODE_MASK & $error_mode;
      }
    }
  }
  
  private function stringCleaner($x)
  {
    if (RequestCleaner::$use_mode & RequestCleaner::USE_HTMLSPECIALCHARS) {
      $x = htmlspecialchars($x);
    }
    if (RequestCleaner::$use_mode & RequestCleaner::USE_HTMLENTITIES) {
      $x = htmlentities($x);
    }
    if (RequestCleaner::$use_mode & RequestCleaner::USE_NL2BR) {
      $x = nl2br($x);
    }
    return $x;
  } // end of arrayHelper()
  
  private function useModesToString()
  {
    $ar = array();
    foreach (array(RequestCleaner::USE_HTMLENTITIES, RequestCleaner::USE_NL2BR,
        RequestCleaner::USE_HTMLSPECIALCHARS) as $mode) {
      if (RequestCleaner::$use_mode & $mode) {
        $ar[] = RequestCleaner::$use_modes[$mode];
      }
    }
    return implode(',', $ar);
  } // end of useModeToString()
  
  public function __toString()
  {
    return "RequestCleaner: examining " . implode(', ', RequestCleaner::$source_names)
      . " Using " . $this->useModesToString()
      . " / Exit Mode: " . RequestCleaner::$error_modes[RequestCleaner::$error_mode];
  } // end of __toString()

  public function __get($name)
  {
    if (array_key_exists($name, RequestCleaner::$cache)) {
      return RequestCleaner::$cache[$name];
    }
    foreach (RequestCleaner::$sources as $source) {
      if (array_key_exists($name, $source)) {
        $val = $source[$name];
        if (is_string($val)) {
          return (RequestCleaner::$cache[$name] = RequestCleaner::stringCleaner($val));
        } elseif (is_array($val)) {
          return (RequestCleaner::$cache[$name] = array_map(array('RequestCleaner', 'stringCleaner'), $val));
        }
      }
    }
    switch (RequestCleaner::$error_mode) {
      case RequestCleaner::NORMAL: return NULL;
      case RequestCleaner::RETURN_FALSE: return FALSE;
      case RequestCleaner::THROW_EXCEPTION:
        throw new Exception("RequestCleaner::__get($name): Value Not Defined");
      default:
        throw new Exception("RequestCleaner::__get($name): ERROR: Value Not Defined / Illegal Error Mode");
    }
    return NULL;
  } // end of __get()
  
  public function __set($name, $value)
  {
    switch (RequestCleaner::$error_mode) {
      case RequestCleaner::NORMAL: return NULL;
      case RequestCleaner::RETURN_FALSE: return FALSE;
      case RequestCleaner::THROW_EXCEPTION:
        throw new Exception("RequestCleaner::__set($name, $value): Setting Attributes Not Allowed");
      default:
        throw new Exception("RequestCleaner::__set($name, $value): Setting Attributes Not Allowed / Illegal Error Mode");
    }
  } // end of __set()
  
  public function __unset($name)
  {
    switch (RequestCleaner::$error_mode) {
      case RequestCleaner::NORMAL: return NULL;
      case RequestCleaner::RETURN_FALSE: return FALSE;
      case RequestCleaner::THROW_EXCEPTION:
        throw new Exception("RequestCleaner::__unset($name): Unsetting Attributes Not Allowed");
      default:
        throw new Exception("RequestCleaner::__unset($name): Unsetting Attributes Not Allowed / Illegal Error Mode");
    }
  } // end of __set()
  
  public function __isset($name)
  {
    if  (array_key_exists($name, RequestCleaner::$cache)) {
      return TRUE;
    }
    foreach (RequestCleaner::$sources as $source) {
      if (array_key_exists($name, $source)) {
        return TRUE;
      }
    }
    return FALSE;
  } // end of __isset()
}

// end class definitions

?>

Why did I post this dramatic exposition of Programming Prowess?

No particular reason, just felt like it.

Why I hate Idea Men

2008-08-31T14:56:00.003-06:00

Ever shoot a rifle and miss the target? Well, if you haven't here's how you miss: you're aim is off by just a little bit.

The same thing happens when you're building software - or anything, for that matter.

I've spent about 40 years building systems of various kinds - mostly software implementations to 'solve' some problem. The process is pretty straightforward and goes like this:

1. Try to figure out what you're going to do.
2. Pick the most important parts and the most important actions and interactions between the parts.
3. Pretend that is all there is in the world and build some software which mimics everything you've thought of. By the way, these parts, actions and interactions are the 'model'.
4. Test it until it works well enough to do the job.
5. Knowing what you now know, go back to step 1 and do it all over again.
6. Repeat step 5.
7. Repeat step 6.

You get the idea.

Nobody is ever right the first, second, third, or even the 'last' time.

Models aren't reality - we just work on them until they are 'close enough'.

What's this got to do with Idea Men?

Idea Men do parts 1 and 2, then they get somebody else to do step 3.

Then they blame the guys who built the thing in step 3 because they don't want to do step 4.

Finally, they won't do step 5 because they're 'right' and the guys who built it are all incompetent and that's the reason it doesn't work like they said it would.

Then the Idea Men get promotions and raises.

Crap

Singleton Patterns in PHP 5

2008-07-01T09:58:00.007-06:00

The Singleton Pattern is one of the few things I really like from Design Patterns. While the original Gang of Four book has some very good stuff in it, later works seem like Software Industry Marketing Fluff [SIMF] - but I've already bitched about that.

The Singleton is GREAT because it can replace Global Variables - if not everywhere, at least in a lot of important cases.

In case you don't know what a Singleton is, it's an Object which has only one instance. Ideally, you would like to write something like

$foo = new Thing();

and get a reference to the one instance of Thing.

The "normal" way to implement the Singleton Pattern is to hack up the class definition in an unnatural way:

Disable the normal constructor by making it private
Define a class variable to hold the One Instance
Create a different public function which is used to instantiate the first instance and return it on all subsequent 'instantiations'

This is because none of the Object Oriented Languages directly support Singleton - except Ruby which has a Singleton base class so you can create them by normal inheritance.

I was so taken with Ruby's Singleton base class, I decided to write one in PHP 5.

Drum Roll........

Didn't work. Failed Miserably.

It turns out that in PHP 5.2 static class variables [that is, Class Variables] are always in the base class. I won't go into that here. Instead, take a look at this test I wrote.

Anyway, just because we can't build a base class for Singleton Objects, doesn't mean we can't clean up their creation. Here is an example of a pseudo-Singleton which allows me to write

$foo = new Thing();

and have a $foo object which acts just like a Singleton even though it has many instances.

The reason this works is that we don't need a single instance of the Object. What we need is a reference which returns the common data, where the methods act on the common data, where modification of the common data is seen by all existing references.

The Pseudo-Singleton looks, acts, and smells like a Singleton, except it is more convenient and natural to use.

The only thing better would be a 'singleton' keyword prefix to use in 'class' definitions.

Reality and Stuff Like That

2008-06-21T22:14:00.003-06:00

Believe it or not, I spend far too much time thinking about what is real and what isn't.

It seems a lot of people seem to think that there are lots of realities and lots of equally valid versions of Truth. This seems like fuzzy thinking to me. So fuzzy that it just doesn't make any sense. [where are you going to put all these versions of Truth - if there isn't a universal Truth which contains them all?]

Anyway, here's what I've come up with: (nothing original here - much stolen from Lao Tzu's work)

The first thing to be clear about is 'how do we think'. 'Cause if we understand that, then we can understand what we think is 'real'.

As nearly as I can tell, we start out by classifying stuff as 'this' and 'everything else'. Some people call this 'dualistic thinking', but my opinion is that it's simply a way to simplify the cacophony of sensory input we are inundated with. It also lets us find 'patterns' so we can form 'habits' to simplify plodding though daily life.

Anyway, when you add some properties to these 'classifications', you get a 'model'. It's a model because the 'thing' has behaviors associated with it. Like a window lets in light and a tea pot holds tea.

The BIG CATCH is when we confuse the 'model's we construct with the real world we live in. The 'model' world is really an illusion we've created - but which we treat as real.

What is 'reality'? That's tough because we're so used to classifying, finding patterns, and creating models that we aren't equipped to perceive - let alone process - what's actually out there.

Anyway - that's what I think is going on. It also answers the quest of why we argue so much:

we're all living in different illusions - not different realities - and our illusions are only loosely based on 'reality'.

There are a lot of conclusions you can draw from this - but this is plenty for now.

ORM's or not?

2008-04-23T09:11:00.003-06:00

In my quest for a CMS I can not only stand but want to embrace, I've spent the last couple of weeks messing with Drupal.

Drupal is impressive - at the very least in scope and participation. I like quite a bit of it. In contrast with Rails, Drupal releases seem to be Designed rather than Grown - which is a very strong plus. [I'm not sure I can stand it, but that's another story.]

But I'm not writing to Praise Drupal, but to make an observation about Object Relational data Mappers.

Drupal through 6 doesn't have an ORM. They have two 'database adapters' and opt to use SQL. Most of the SQL used for CRUD operations is standard, so that works just fine. I'm told there is discussion in the development forums about stuffing a layer of abstraction in the database code - which I read as 'create an ORM.'

Looking over the landscape littered with the corpses of ORMs in almost every conceivable (computer) language, should give pause - at least it does to me.

Why do we ORM?

Well, we have to have the data tables in the Database match up with the Attributes in the Objects, don't we?

Well, No we don't.

There's no reason for the Data Tables to model our 'business process'. That's a knee jerk reaction taught as gospel in C.S. courses.

Couple of Facts:

Databases are really good at holding, retrieving, searching, and sorting vast quantities of data. Programming Languages aren't.
Programming Languages are really good manipulating, displaying, inputing, validating, mashing, spindling, and folding data. Databases aren't.

Now for a Web Site, much of the content doesn't have a long life, is retrieved by simple, well known searches, ordered by well known sorts, and is displayed in small chunks.

It seems to me that there much of the data used in a web site could be modeled exclusively in the Programming Language and then serialized in XML or JSON for storage in the database. That separates the representation of the data in the application from the representation in long term storage and obviates the need for an ORM.

I like that idea so much, I'm building one now just to see how it works in practice.

Stay Tuned

DRY versus WET

2008-03-28T09:17:00.005-06:00

You've got to have either a Snappy Acronym or Fancy Vague Phrase (FVP) in order be credible. So when I decided to write about what I think is wrong with DRY, I came up with 'DRY versus WET' - but I didn't know what WET meant.

DRY - Don't Repeat Yourself - is one of the Mantras of the Current Age of Programming Wisdom.

In case you don't know what a mantra is, it's a sound or phrase which you repeat over and over again while meditating. Meditating is something you do with your mind which brings peace and quiet . . . by avoiding all thought.

The problem I have with DRY is that you have to Think in order to write Good Code. In other words, you need Well Examined Thought - that's where WET comes from. [So now I've got an Acronym that Actually Means Something].

For Example:

Take a look at the Rails source code to see what over-DRY-ness does. After a while you'll see that (almost) every piece of code which could be repeated is turned into a method call. Even stuff which isn't more than a fraction of a line! This is Really DRY - so It Must Be Good. Right? Wrong!

This is Spaghetti [see The New Spaghetti for more ranting . . I mean 'detail'].

Historical Note: Spaghetti code originally referred to the overuse of GOTO (or branch) statements in FORTRAN or Assembler programs, but any program which has a high ratio of Control Transfer statements to Useful Work is really Spaghetti - because it reads like all the code was written on the sides of noodles and then all mixed up.

So . . . Don't DRY up. Get WET!

Think it's worth doing a book about?

The New Spaghetti

2008-03-13T17:18:00.004-06:00

Back before Structured Programming - which everyone now thinks is as natural as breathing - we had a thing called 'Spaghetti Code'.

You wrote Spaghetti Style by using lots of conditional and unconditional GO TO statements to re-use every bit of code you'd written. It was the privative precursor of DRY [Don't Repeat Yourself].

In those days it kind of made sense - for a couple of reasons:

1. we didn't know any better

2. memory was tiny - we measured big machines in KiloBytes and wrote programs using 'overlays' [in case you don't know, an overlay is a chunk of executable code which 'overlays' another chunk in memory. We used to do that because (a) we didn't have enough memory to do the job and (b) there wasn't any automatic memory management]

The result of Spaghetti Style is that nobody could ever figure out how the programs worked and modifying them was next to impossible.

What brought this up?

I've been trying to 'learn Rails' - which is a pretty large Ruby application. Rails has close to 10,000 methods defined, about 1,500 classes, and God only knows what else. The Vocabulary is huge - and so it's a daunting task to say the least.

On top of the huge vocabulary, there it's DRY.

That seems to mean that every chunk of code which is used two or more times is ripped out and turned into a method. The result is that in order to do anything, somewhere near one and a half gazillion methods are called. In order to figure out how almost anything works you have to work through a call stack which is makes the old FORTRAN Spaghetti look like a 2nd grader's maze.

Why is this happening?

I'm guessing, but I think there are three factors driving this result:

1. DRY - This is a misunderstanding. DRY is a hip way of saying 'we want to maintain a single point of control so that we don't have to repeat bug fixes in the code'. DRY shouldn't be a mantra, because it's not a Principal of Good Programming. It's a method to implement an idea which is good programming.

Prior to formulating 'Single Point of Control' as DRY, programmers used some judgement in the size of the methods/functions they wrote. Typical size was 5 to 25 lines or so - enough to do useful work and small enough to be easily understood. Rubyists seem to be addicted to one-line methods which call other methods which call other methods which ... - but they don't Repeat!

2. Agile Development - This seems to go hand in hand with Test Driven Development. Both sound like good ideas, and they seem to work well for 'throw away' applications - which is what most web sites are. After crawling through the Rails Association Code, I've come to the conclusion something critical has been lost in the Agile-ness of the Development: There's no Design Phase.

Test Driven Development has proven you don't have to have an overall plan to build code which works. You can build any twisted mess and if you continuously test, it will pass the test.

But it won't make sense, won't be cohesive, will probably be inefficient, and it will be hell to modify or learn.

Everything has to be designed it is going to work well and be maintainable. That's true for Buildings, Cars, Motorcycles, Bridges, and, believe it or not, Software.

3. Ruby itself.

I love Ruby - almost - because it's a beautiful language and it makes it easy to otherwise hard things. The fact that Ruby is a scripting language means we get instantaneous feedback - and I love that too.

But both of those things - along with Ruby's openness which allows all kinds of self-modifying code [which seems to be confused with meta-programming] makes it easy to get the illusion of 'doing something' when the programmer is actually just churning different implementations with little impact on improving the goals of the application being developed.

Does that make sense? If not, read it again.

Activity != Progress

It takes discipline and experience to handle as powerful a tool as Ruby is without creating a mess.

Don't believe me?

Just take a look at the source for Mongrel and compare [or rather Contrast it] with a bunch of the Rails code. Zed Shaw writes awesome, easily understood code. Somebody in the Rails Core doesn't.

Hey Ruby: Where's the Main Program?

2008-02-22T07:40:00.005-07:00

So as I get deeper into Ruby and Rails, I've found I've been going absolutely nuts! trying to understand how the programs work.

I think I finally figured out my problem. (That's right, it's a problem with ME for a change!)

There's no 'main' in a Ruby program.

Most executables - as in /usr/local/bin/foo - are two to ten line Ruby scripts which set up some environment variables and 'require' the 'real' script. The 'real' script usually 'require's a bunch of other scripts, instantiates an object, and calls what appears to be a random method.

That's it.

Step back and contrast this with C, Java, Python*, FORTRAN, COBOL, the Boot Sequence of a computer, ... Need I go on? Everyplace else there is (or pretty much is) a well known function, procedure, 'program', file or something where execution starts for every program which is written. Even Javascript kicks off with an 'onload' function (except in JQuery, but I think that's really a wrapper).

Notice I've left out Lisp, Scheme, Smalltalk and a whole bunch of other languages.

I don't know much about them - from a practical point of view - except that they are interpreted, don't (to my knowledge) have a 'main' entry point, and allow programmers to modify the base system so that it is unique to their program.

Now, I know someone might argue that all Ruby programs start by firing up the interpreter and loading the environment, so I'm wrong - as usual. But that's not what I'm talking about.

'main' is the start of the logic of the program. Knowing this gives (the generic) you a place to start unraveling the logic of a program. This seems like a 'good thing' (trade mark), but is it?

The whole idea of 'main' was really invented so that other programs (loaders) would know where to start executing a program. This is a 'convenience' with compiled programs: programs where the source code is processed into (possibly virtual) machine instructions and then further processed into an executable image saved in a file. This executable image is then loaded, source code agnostically, into the execution environment of a computer and run. It's a simple fact that the 'loader' needs a place to start and it doesn't know from beans about the source code - hence the well known 'main'.

You don't need this for interpreted, dynamic languages, but the S/W community seems to have carried this structure forward.

I think Ruby represents a significant break from this architecture.

Ruby 'programs' seem to be a bunch of symbiotic objects, executing in a common (probably specialized) runtime environment. Various behaviors (programs) can be achieved by instantiating different objects and calling appropriate 'starting' methods. This is similar to, but much more fluid than, writing a program within the constraints of a published API for, say, Windows, or a purchased library.

I like it.

I don't know how to understand it easily yet, nor do I think anyone really knows how to document it yet, but I think it's an advance.

* Python really doesn't have a 'main' either, but it has taken the 'flavor' of 'main' stipulating that code can be conditionally executed if it is 'run' as though it were the 'main' program AND a chunk of code is included of the form:


if __name__ == '__main__':
  some code

This chunk typically goes at the bottom and is encouraged when the code in the file can server either as the basis for a stand-alone program or an importable unit to another OR for testing.

DSL Design - that's Domain Specific Language

2008-01-27T07:32:00.000-07:00

A Domain Specific Language - for those who don't know - is a bunch of functions named so that writing function calls 'reads' like natural language. It seems to be sprouting wildly in Ruby - most probably because Ruby doesn't require parentheses around function arguments and Ruby programmers are kind of rebels anyway.

For example:
rabbit_jumps_in_the_hole :hole_size => 10

Even if you don't understand Ruby Syntax, you know what the function call does.

Think of it this way:

Function/methods are really 'verbs'.
Function Options are 'adverbs'
Object Identifiers are 'nouns'
Object Attributes are 'adjectives'
Class Names are 'class names' [gottcha!!!!]

I think we should think of learning one of these DSL things the same way we think about learning a new Language.

This can be either a good thing or very bad.

Size Matters

Which is easier to learn: A language with 10 verbs or one with 1,000?

Just for the heck of it, I recently tried to get a count of the 'verbs' in Rails 2.0.2. I ran 'egrep -r 'def [a-z]' on all lib directories and came up with:

actionmailer/lib 694
actionpack/lib 1393
activerecord/lib 1134
activeresource/lib 125
activesupport/lib 577

Total 3923

In contrast, the Merb Framework is much smaller:

merb 713
merb.rb 19
tasks.rb 0

Total 732

Of course, this isn't fair because Merb doesn't come with an ORM [Object Relational Mapper library (bunch of database access functions - for those really out of it)] [or Active Record Pattern Implementation, for those . . . - well, you know who you are], so you have to add that in.

But Merb gives you a choice of ActiveRecord - with it's 1,100 verbs; DataMapper - with about 500 methods; or Sequel - with about 600.

So learning Merb should be easier than Rails because the vocabulary is about 1/3 to 1/2 the size.

Synonyms are Bad

Which is easier to Learn: a language with one word for each concept or with two or more?

A programming language or environment isn't meant for composing poetry, novels, or movies. It's supposed to precisely express a procedure. Period. It should be concise. That makes it easier for Programmers to understand.

Case closed. DSL's should be concise, singular, and boring - but very, very accurate.

Corollary:

The Rails Inflector is a mistake in every possible way:

It Expands rather than Tightens the vocabulary of the Rails DSL
It injects confusion because Programmers now have to worry about singular and plural forms depending on context
It doesn't work:
- 'XMLClass'.underscore -> 'xml_class'
- 'XMLClass'.underscore.camelize => 'XmlClass'
- 'slave'.pluralize == 'slaves'
- 'slave'.pluralize.singularize == 'slafe'
It wastes lots of cycles doing it - machine, programmer, and learning

Distance is Good

I'm talking about the distance between words. For example frog is very close to frogs but far from toads. That makes it easier to tell a frog from a toad in print than in real life.

Good DSL design should not only use expressive and concise identifiers, but should also keep them far apart, especially when the referents do significantly different things.

Again, picking on Rails, the methods update_attribute(attribute) and update_attributes(attributes) are very close together, but one bypasses attribute Validation. Can you tell which one by the names? Don't you think it's important to know?

DSL is Not Documentation

Most DSL seem to grow more or less organically. The Ruby universe is filled a lot of apparently useful packages with virtually no documentation. Almost all of them have fairly reasonable API documentation - which allows 'one' to learn what each of the 'verbs' in the DSL do, but that's like learning to drive a car by reading Glossary of the Parts! It Just Don't Work.

It's hard as hell to learn a system without some sense of what the thing is supposed to be doing and how it's put together.

Don't belive me?

Figure out a Car from stuff like this:

Wheel - 1. circular object in contact with ground; 2. circular object interfacing driver to directional controls.
Nut - 1. Device for attaching wheel; 2. driver in other automobile; 3. nutritious snack
etc.

That's API doc and that's what you've got when all there is is the DSL.

'nuff for now

Is Java Bad for You?

2008-01-25T09:36:00.000-07:00

How do Heavily Constrained programming environments - such as Java, C#, and friends - effect our thinking and creativity?

I think the goal of heavy constraints and requirements started out as a way to get better code by automatically checking as much stuff as possible mechanically. It all started with compile time type checking and has extended into things I don't want to know about.

Anyway, the result is that it's hard to write code with these tools. From what I hear - and I don't do Java, C#, and friends - the 'programmers' spend most of their time figuring out what API's and Design Patterns to use. I don't find that fun at all.

I usually spend most of my time trying to better understand the problems I'm trying to solve and creating software structures which mimic the nature of the problem. After I've coded and tested one of these structures, I call it a 'solution'.

In other words: I don't use Design Patterns and don't think in terms of API's.

Does that make me a dinosaur?

I don't think so.

I gravitate toward unrestricted programming environments. My first introduction was SCO Xenix around 1986 or 87. I became really excited as I realized how easy it was to do mundane tasks by stringing filters together in pipelines. I could accomplish more useful work in 1/10th the time [or so it seemed] than I could writing special programs to do the same thing.

In addition to being faster, it was more fun. I spent more and more of my energy solving problems rather than conforming to code writing rules.

The same thing happened when I discovered Python - and now to a similar extent Ruby. Scripting languages with good support for dynamic strings, arrays, hashes and objects are wonderful. They handle all the details of what I need - as a coder - to do the job.

How do I keep from hurting myself when the Programming Environment doesn't keep tabs on me?

Well, it's not a problem. I just test as I go and keep rewriting my code so it works, is more succinct, and tighter. [I guess you call that Refactoring now - we used to call it rewriting]

The facts are that Anyone can write bad code - and restrictive frameworks don't stop them. Anyone can also write good code - if they take the time to learn how and pay attention to what they are doing. And restrictive languages don't help with that either.

When I need a solution - I just think one up (or two or three) up and try it out. It's easy to write the code, change it, test it, and refine it. In a verbose, API laden environment like Java, that cycle isn't so easy - or at least is a heck of a lot more verbose.

I suspect that restrictive environment programmers get dulled down by the drudgery of just writing the code and learning all the API's. There is so little room for creativity that they lose it - creativity is something you have to practice and cultivate.

As a result, they get used to solving problems by applying packages and patterns. They don't really design: they apply old designs to new problems and hope that they work. [BTW, that's the reality behind Anti-Patterns]

So, I guess it makes sense: if everything you do is a cut-and-paste of something somebody else thought up, you would apply that to Design as well.

God, that's boring!

If I'm write, then Design Patterns are a result of Boring Programming Tools which create Bored, Dull Programmers and more Bad Code.

I don't think that's a good thing.

What do you think?

Code Efficiency in Ruby

2008-01-09T11:08:00.001-07:00

I got interested in the qualify of Ruby code in Rails when I noticed what appears to me to be a useless method in the ActiveRecord code. Specifically, ActiveRecord::Base.save is a public method which calls the private method ActiveRecord::Base#create_or_update. That doesn't make a lot of sense to me, because it could be replaced by making 'create_or_update' public and then aliasing 'save' to it.

So I decided to check to see what the superfluously method call costs.

I wrote a test which performed a simple task [incrementing an instance variable by a random number between 1 and 10] using five (5) different ways of accessing the instance variable and invoking the action.

The class definition is at the bottom of this post.

I then ran these methods 10,000,000 times using four different ways of invoking the methods:

directly calling the methods - e.g. foo.inc_instance_variable()

invoking via the method's 'call' attribute - e.g. foo.inc_instance_variable.call()

invoking via the method via 'send' - e.g. foo.send('inc_instance_variable')

invoking via 'eval'ing the string - e.g. eval 'foo.inc_instance_variable'

The Precent Results are simply the run time divided by the minimum run time for all tests converted to a percent increase.

Here's the summary:

Invoking via an Alias doesn't cost anything

unnecessarily accessing an instance Variable via an accessor slows down about 20%

The unnecessary method/function call slows down by about 25%

Combining the unnecessary call with accessor access slows down about 45% - so the effect is linear

Invoking by the 'call' method slows it down by about 13%

Using 'send' slows down about an additional 45%

Using eval slows the process down by something on the order of 400%, but the effect is not linear, so 'eval' must be doing some additional mucking about.

So, what's the point? None, if you're satisfied with glacial execution speeds.

On the other hand, it's something you should know if you are writing critical code and have to make choices about how to implement it.

As usual, your mileage may vary. The full program code is at http://www.clove.com/downloads/method-call-timing-tests.rb.

Here are the detailed Percentage Results

Percent Results for direct method of invocation
foo.inc_var_as_instance                   0.99
foo.inc_var_as_instance_alias             0.00
foo.inc_var_as_method                    19.89
foo.inc_var_as_func_and_instance         25.28
foo.inc_var_as_func_and_method           45.17

Percent Results for call method of invocation
foo.inc_var_as_instance                  14.06
foo.inc_var_as_instance_alias            12.93
foo.inc_var_as_method                    32.53
foo.inc_var_as_func_and_instance         42.19
foo.inc_var_as_func_and_method           59.52

Percent Results for send method of invocation
foo.inc_var_as_instance                  42.05
foo.inc_var_as_instance_alias            46.02
foo.inc_var_as_method                    60.65
foo.inc_var_as_func_and_instance         75.57
foo.inc_var_as_func_and_method           94.03

Percent Results for eval method of invocation
foo.inc_var_as_instance                 393.89
foo.inc_var_as_instance_alias           410.94
foo.inc_var_as_method                   420.03
foo.inc_var_as_func_and_instance        458.10
foo.inc_var_as_func_and_method          577.84

Here are the raw timing results:

Result using Direct Calls
foo.inc_var_as_instance   7.070000   0.040000   7.110000 (  7.306318)
foo.inc_var_as_instance_alias   7.000000   0.040000   7.040000 (  7.155283)
foo.inc_var_as_method   8.390000   0.050000   8.440000 (  8.585970)
foo.inc_var_as_func_and_instance   8.780000   0.040000   8.820000 (  8.950350)
foo.inc_var_as_func_and_method  10.160000   0.060000  10.220000 ( 10.378127)

Result using .call
foo.inc_var_as_instance   7.990000   0.040000   8.030000 (  8.204646)
foo.inc_var_as_instance_alias   7.910000   0.040000   7.950000 (  8.061320)
foo.inc_var_as_method   9.280000   0.050000   9.330000 (  9.502992)
foo.inc_var_as_func_and_instance   9.940000   0.070000  10.010000 ( 10.249010)
foo.inc_var_as_func_and_method  11.170000   0.060000  11.230000 ( 11.439927)

Result using 'foo.send '
foo.inc_var_as_instance   9.950000   0.050000  10.000000 ( 10.220120)
foo.inc_var_as_instance_alias  10.220000   0.060000  10.280000 ( 10.462063)
foo.inc_var_as_method  11.250000   0.060000  11.310000 ( 11.514329)
foo.inc_var_as_func_and_instance  12.290000   0.070000  12.360000 ( 12.583919)
foo.inc_var_as_func_and_method  13.590000   0.070000  13.660000 ( 13.891333)

Result using 'eval '
foo.inc_var_as_instance  34.550000   0.220000  34.770000 ( 35.387702)
foo.inc_var_as_instance_alias  35.740000   0.230000  35.970000 ( 36.670451)
foo.inc_var_as_method  36.390000   0.220000  36.610000 ( 37.296297)
foo.inc_var_as_func_and_instance  39.050000   0.240000  39.290000 ( 40.033456)
foo.inc_var_as_func_and_method  47.420000   0.300000  47.720000 ( 48.638035)

Here's the class definition:

class Foo
attr_accessor :var

def initialize
 @var = 0
end

# access the instance variable directly
def inc_var_as_instance
 @var = @var + 1 + rand(10)
end
# access instance variable directly, but us an alias
alias_method :inc_var_as_instance_alias, :inc_var_as_instance

# access instance via accessor method, even though inside class instance
def inc_var_as_method
 self.var = self.var + 1 + rand(10)
end

public
# add an additional method call to accessing via direct access to instance variable
def inc_var_as_func_and_instance
 inc_var_as_instance
end

# add an additional method call to accessing via accessor
def inc_var_as_func_and_method
 inc_var_as_method
end
end

Design Patterns and the Fall of S/W

2008-01-05T11:19:00.000-07:00

One nice thing about having a blog nobody reads is that I can say anything I want without worrying about it biting my butt.

I hate Design Patterns.

It's that simple.

I hate the guys who promote them.

Most of all, I hate the s/w industry - especially the programmers - for being duped by these guys.

And I'm qualified.

I received an engineering education and am a self taught computer 'something'. I'm not exactly a programmer, although I've written an awfully lot of code in a variety of environments - all the way down to machine code on microprocessors up to hokey database/user interface stuff. I've done device drivers and created my own little languages using lex & yacc and 'in the raw' in C, Python, and awk. And I've been doing this over 40 years - so I've earned the right to be a grouch.

The Design Pattern guys are the current generation of Yordon (sp?), Codd(sp?), and Bouch (sp?): Consultants who watch other people create software while telling them how to do it right. None of them actually do anything, but the sure create a lot of bad advice.

I remember buying three (3!) books by Peter Codd(sp?) on Object oriented programming and design only to find out that he admitted that he didn't really know anything about it. The idiot had gotten excited about the idea, so he and his group spent a year puttering with it and writing books - probably giving lectures and doing expensive consulting with BIG companies at the same time.

It takes years of experience doing something to understand the concepts. [things move quickly, but our brains take a while to catch up]. Hell, it takes 10 years plus to design, implement, and knock most of the bugs out of any programming language.

The Design Pattern guys are the absolute worst! The claim that they are defining these things to add clarity to the software process and they they write the vaguest crap imaginable. Don't believe me?

Grady Bouch: "In the world of software, a pattern is a tangible manifestation of an orginization's tribal memory." - CoreJ2EE Patterns (introduction) [clipped from PHP 5 Objects, Patterns, and Practice - by Matt Zandstra]

Merrian Webster Dictionary: "1. an ideal model. 2. something used as a model for making things. 3. Sample"

Which is clearer? If you like Bouch - then you need to join a consulting company and stop pretending to write code.

I'm starting to froth at the mouth, so it's time to simplify things. Here's a simple procedure to see for yourself.

1. go to a book store and pick up one of Martin Fowler's many books on patterns. WARNING: Do not buy the book.

2. Select a pattern at random and read the first paragraph describing it.

3. Answer yourself honestly: Do I know what this 'pattern' is well enough to describe it in a single sentence?

if No - read the rest of the description and try again

if still No - replace book on shelf.

if Yes, please send that sentence to me.

Thanks,
Mike

What's the Best Web Site Builder

2007-11-11T08:49:00.000-07:00

I don't think it exists yet.

There is a parallel between Web Site Frameworks (and friends) and Operating Systems and I think we can build better web site development systems if we're aware of it.

About OS's (at a really light level)

Operating systems really stated out as Monitor programs which allowed a 1950s and '60s 'programmer' to load and run one program at a time on the monster sized, minuscule capable early machines. They sucked, but it was better than programming in machine code.

The big advantage Monitors gave was that if you upgraded the hardware, you reflected the change in the Monitor and your old programs still ran. MS-DOS and CP-M were really Monitors, not operating systems.

Then smart people developed the idea of an Operating System. One problem with Monitors is that they didn't have any way to recover control of the hardware when the application program went into a infinite loop or some other destructive mode. OS's had an executive module which allowed the program to run. It regained control periodically and decided whether or not to kill the program or let it run. In other words, you had to get permission from the OS to run your program. It sat there like a despot, completely controlling what you could and couldn't do and how long you could do it.

To get Back to Web development, I think we're at the 'early-ish OS stage'.

We've already gone through the Machine Code and Assembler Language stage with raw HTML and doing everything with Tables. Monitors roughly correspond to writing everything by hand - with or without a tool like Dreamweaver - one site at a time.

Drupal, Rails, Joomla!, and friends look very much to me like early OS's. When a request goes to the server, it is caught by the framework's machinery. The framework (read OS executive/sultan/king) then decides what to do with it.

So What's the Point?

What's the most successful OS model of all time? UNIX!

Why?

UNIX gave Choice to the Programmers and took it away from the OS Designers.

UNIX was a fundamentally different from the other OS's. IBM and everybody else put everything that a good police state needs to control it's citizens inside the operating system where nobody could get at it except the privileged few. UNIX didn't.

The UNIX system consisted of two parts: the kernel, which provides services for it's citizens and userspace, where things are done.

The kernel supports and enables. For the most part, it doesn't control. All the Executive stuff is in userspace which means: it can be replaced by the people Using the machine if it doesn't do what they want.

This is really important so I'll repeat it. The kernel supplies services, but doesn't do anything. It just sits and waits - like a good robot. All the action is in userspace and it can be replaced if you want something different.

All the web frameworks I've looked at over the past year and half have the Executive function embedded in the framework. That is, when a request comes it, it's received and processed by the framework. Right now in Web Development, if you want to do something which is not directly supported by your framework, you have three choices:

hack together some lame work-around
don't do what you want
get another framework

It's just like the pre-UNIX days.

When UNIX came in, it became possible for the Programming Community to build lots of alternate Executive function implementations as drop in replacements for the stuff which shipped with the system. Free Market!!! That's why UNIX is a great development environment and why it makes such a great server environment. The Programming Community developed a lot of stuff and kept the stuff which works.

That couldn't have happened if we'd had to change kernels for every alternative design - or it would have taken a gazillion years longer.

The Pitch

I think we need Web Development kernel. I'm working at spec'ing one out. I call it uWeb (pronounced 'you Web')

Me and the Leopard

2007-11-09T08:06:00.000-07:00

I installed Mac OS X 10.5 - aka Leopard - last Saturday (or was it Sunday) and just now finished reverting to 10.4.10. Boy oh Boy, has it been non-Fun.

Well, it was really my own fault. I know better than to install the initial release of anything, but, I'll admit it, I got lulled into wanting it - salivating even - by listening to Steve the Jobs extol Leopard's features. He's really right: they are really neat, look great, and will make using the machine so much easier. So when Leopard came out, I pounced.

Anyway, skippin’ the boring details:

Printing to my HP All-in-one died dramatically
Time machine doesn't work across a LAN w/o OS X Server - for another $500
'admin' accounts spontaneously became 'standard - Twice

So I got scared and spent the next three days reverting back to Tiger. (I finally did an Erase and Install and reinstalled/rebuilt everything)

I'm waiting for 10.5.1, and then I'll try it again - maybe.

Design Patterns?

2007-11-03T10:24:00.000-06:00

I was asleep in the country when Design Patterns became popular. It seems now that unless you've had a 'pattern implant' you're not qualified to solve problems.

This came as a shock to me because I've been analyzing and solving problems for a long time: I escaped Engineering school in '68, '70, and, finally for good, in '72. We didn't have 'design patterns' then. We had principles, experience, and knowledge.

So what are these things? Are they a good thing or not?

According to the Wikipedia article on Design Pattern (computer science), a design pattern "... is a description or template for how to solve a problem that can be used in many different situations."

So far that sounds good, but then I start getting confused:

design patterns are not algorithms because "algorithms solve computational problems." I didn't know that. I thought algorithms were systematic, iterative procedures which did something - like copy a buffer from one place to another or to find something.
design patterns are not "architectural patterns" because they have a "different scope."

Then I read some stuff by Martin Fowler describing the Active Record pattern. The thing that came across most strongly is vagueness. Everything is so indefinite and "could also be ...".

It all became crystal clear!

Design Patterns are the current Holy Grail of the Software Methodology Guru.

As long as I've been beating on computers, there have been Great Gods of Software Design. I used to read their stuff and think I was doing everything wrong because I wasn't doing it their way. So I'd try it and it didn't work very well for me (why later).

The Software Methodology Gods are really OK guys, but you have to judge their gospel within its context. In other words: what do these guys really do?

One thing the don't do is actually build the kinds of systems they tell you how to build. Why? Well, they don't have time. They spend their time watching other teams build things and writing and preaching about what they've observed. At least that's what they are supposed to be doing. [And the reason it didn't work for me is that I've never worked on a team. All my stuff was in-house, solo or with a partner, limited scope, and very limited user base. Their experience did *not* apply! see Joel on Software, chapter twelve "Five Worlds"]

So these guys' main business is giving advice.

I'm sure a lot of the advice is pretty good - it has to be if they get from watching teams that do good work. But that doesn't mean it's right 100% of the time and in all cases. Anyway you can't really tell if it works, because there's no way you can compile and execute vagueness.

OK, I'm a little harsh. It's just that if the advice is vague, then it's usually pretty easy to win the postmortum no matter how things work out - success or failure. If it worked, it's because you followed my advice. If it failed, it's because you didn't. Vaguness covers or excludes a whole range of implementations.

Anyway, what happens is this: These guys sound so confident that I used to get lulled into thinking that they're right and I was wrong.

What's the result?

I'd read their stuff and instead of trying to figure out if it is worth doing in my situation, I'd spend all my time trying to figure out what they're talking about and how to change what I was doing to be right - according to them.

I was working my brain, but not on the right problem. (By the way, would you be interested in this really great vacuum cleaner we have on special today?)

So, here's what I think Design Patterns are good for:

ideas - look them over and see how somebody else does it
obfuscation - if you know enough of them, you've got a great cryptic language you can use to blow smoke in the other guy's face
writing books and articles - "Design Patterns for ________". Just fill in the blank and you're all set for fame and fortune
getting consulting gigs from non-technical managers
giving courses to programming teams

One thing they won't do is make mediocre and poor programmers productive.

So what am I going to do now? Well, I'll read the book. There must be something useful in there. But I'm not going to shut off my brain when I design something.

Why MVC is Wrong

2007-10-28T15:43:00.000-06:00

First of all: MVC is Right because the model separates problems into three relatively independent functions.

But, MVC is wrong because it's being used as though it is the Only way to do things.

It's Not.

The problem is that Presupposing or Requiring an MVC model interfers with the Design Process.

Here's the best way we've found to design and construct systems:

1. figure out what is to be done and define it precisely and succinctly

2. define a collection of tasks which can be performed in series or parallel where the results can be either combined or fed to successive tasks which produce the desired result. Ideally, these tasks can be defined as 'black boxes' - to use the term I was taught back in Engineering school.

3. Build the boxes, test, deploy and celebrate.

In my experience, every field - Engineering, Computer Science, Economics, etc etc - has rediscovered the same method and principles - but uses different terminology.

In CS, black boxes were 'objects'. Now they are Design Patterns. We used to talk about Heirarchic Decomposition, Information Hiding and Structured Programming. In client-server designs, we talk about clients, servers, and protocols.

It's all the same stuff.

1. We figure out what we want to do.

2. Then we chop it into pieces small enough to build.

3. Then we define some protocol to get inputs into and outputs out of each piece and use them to wire the whole thing together.

MVC Screws this up mostly because it puts the Chopping before the Analysis. This Adds complexity to the solution rather than taking it away.

For Example:

Suppose I have a semi-static web site. By that I mean, nothing changes all that much, but the owner would like to modify something on a couple of pages every once in a while.

I can Make this MVC, but why bother? There's nothing to Control. All I have are Views. The only 'model' I have is a chunk of Text. Most of my pages can look like:

<html>
etc
<?php include('owner-content'); ?>
etc
</html>

If I have a way for the site owner to update 'owner-content' - possibly a simple update page so she can edit the page on her home machine and then download it to the site - then I'm done.

I don't need MVC partitioning of this problem unless I want to use a prepackaged system.

The obvious answer is there isn't any need for all that machinery to solve simple problems.