Sunday, April 21, 2013

I just realized that I spelled "unobtainabol" wrong when I chose the URL, so I'm moving to a URL with the correct spelling. The new blog is here.

Dave's Foreign Data --Introduction

[Oops. Just realized that I spelled "unobtainabol" wrong when I chose the URL, so I'm moving to a URL with the correct spelling. The new URL is here. The new blog is here]

Postgresql (aka Postgres) is a really impressive open-source project. It is a relational database management system that supports a large subset of standard SQL and also has lots of cool extra features as well as amazing extensibility. For example, you can write stored procedures in about a dozen different languages. You can create new aggregate functions, types and index implementations among other things. The latest versions also have a foreign-table interface which lets you create a table in a Postgresql database which is a sort of alias for data that exists somewhere else.

One of the cool features that Postgres has is table functions. You can define a function that returns a sequence of rows rather than just a value. You can call a table function in a contexts where a table is expected as long as the table is not being modified (as in an update statement, for example). The function then gets called when Postgresql needs rows and it returns the results of the function.

 I wrote a foreign-table interface for Postgresql a few years ago before Postgresql had a foreign-table feature so I had to fake it by writing a plugin. I'll call my project dfd (Dave's Foreign Data) to distinguish it from the new official Postgresql feature fdw (Foreign Data Wrappers). Dfd is different from fdw in a number of ways. First, dfd only works for a foreign SQL database. It would be a major effort to make dfd work for a non-SQL data source. By contrast, fdw is designed to be more general and there are wrappers written for such things as flat files and web sites. Second, when you install a foreign database in dfd, it automatically creates Postgresql foreign tables for all of the tables in the foreign database (because dfd was intended to let Postgresql be the query front end for a SQL data warehouse). In fdw, the user has to manually create foreign tables. Third, dfd contains optimizations that are not available by default or not even possible with fdw.

Dfd works as follows. There are two table functions
dfd_call(connection text, query text)
dfd_stub(connection text, tablename text)
The dfd_call() function sends a query to a remote database and retursn the results like any normal table function. This function is used for implementing foreign tables as well as allowing the Postgresql user to send pass-thru queries to the foreign database. The dfd_stub() function is a dummy that should never be called. It does not do anything other than report an error.

To represent a foreign table, dfd creates a view that calls dfd_stub(). For example, suppose the foreign database db1 has the following table:
create table foo(x int, y text)
then dfd links to this table by creating roughly the following view:
create view foo as
  select * from  dfd_stub('db1','foo')
   as tmp(x int, y text)
The part following the second "as" specifies the row type of the foreign function. This is required by Postgresql when a table function was not declared with a row type (as an aside, this is an interesting way to support late binding in a language with static types and I'll probably steal the idea for Unobtainabol).

Now, since dfd_stub() is a dummy function, you can't call the view as it is. Dfd contains a pluggin that goes into Postgresql between the parsing phase and the planning phase. This plugin looks for calls to dfd_stub() and modifies the syntax trees that contain such calls. One of the things it does is replace the call
dfd_stub('db1','foo')
  as tmp(x int, y text)
with a call like
dfd_call('db1', 'select ... from foo ...')
  as tmp(...)
Where the ...'s represent whatever special code is generated to implement the foreign call. For example, if x is not used in the outer query, it will leave x out of the projection list and the corresponding type list.

The first implementation of dfd only used a plugin (written in C) and some PL/PGSQL code. Unfortunately, it turns out that Postgresql table functions don't return a row at a time. In order to support cursors, Postgresql evaluates a table function all at once, storing the rows in a buffer then returns them one at a time as required by the plan executor. In an application like dfd, this creates too much first-row latency to be tolerable, so we had to modify Postgresql to return rows one at a time for table functions.

Of course modifying the Postgresql source code causes problems whenever you want to upgrade to the next version of Postgres, so for the next version of dfd, I'm hoping to exploit the fdw mechanism. To do that, I'm going to have to find a way to do some optimizations that fdw does not support.


Syntax of Unobtainabol

Unobtainabol doesn't have syntax (I'll explain how that is in a later post) but it's hard to discuss langauge features without giving examples, so I'm going to present here a sort of default syntax for discussing the language.

In many languages, there is some specific syntax for declaring a program (Pascal), or a program has to be declared in functions (C, C++) or a program consists entirely of classes (Java) or of packages (Eiffel). These kinds of restrictions don't really seem to add anything of value, so Unobtainabol opts for the simplicity and flexibility of the Lisp model: a program is just a programming language expression.

Of course this means that things like function, class, or module declarations are expressions. In Unobtainabol, everything is an expression. There are no special syntactic categories for declarations, statements, or other phrases that are not expressions.

The entire purpose of this syntax is to explain the language, so I'm going to use a syntax that is easy to to read and figure out for someone who doesn't know the rules of the language --one inspired by SQL.

 There are two basic forms of expression, clauses and terms. Syntactically clauses are used for things that would be statements in other languages and terms are used for things that would be expressions. But I'm not calling them statements and expressions because everything in Unobtainabol is an expression.

Terms look like expressions in common programming languages:
x
x+y
f(x,y)
x := y
 but they can also use keywords as in SQL
x between y and z
x in y
A function call with the traditional parenthesis always evaluates and dereferences all of its arguments and passes them by value. So if the argument is a variable, the value of the variable cannot be changed by the call. Of course if the variable refers to a changeable object, the object may be changed. There are also procedure calls that are syntactically different, using curly brackets:
p{x,y}
This kind of call can use any kind of parameter-passing method so the user can't make any assumptions about it without looking at the signature. Unfortunately, this is a novel syntax that breaks the rule about programs being easy to understand for people who don't know the language, but at least the odd syntax will warn them that there is something unusual going on.

A term can be made into a clause by putting a semi-colon after it:
f(x,y);
p{x};
x := y;
This becomes important because there are constructs that allow a list of clauses and in such a list, you need a syntactic way to tell when a term ends. The other clauses all look like statements do in languages that fully brackets with keywords

if x < y then
  y := f(x);
  x := x+1;
elif y < x then
  x := f(y);
  y := x+1;
else
  g{x,y}
fi
do while x < y;
  x := 2*x;
od
do for var i := 1 to 10;
  p{x,i};
  y := y+i;
od
Declarations are usually terms that begin with a keyword. Since they are terms and they usually appear in a list of clauses, they are almost always followed by a semi-colon:
do while x < y;
  var i:integer;
  i := 2*x;
  x := i;od
The type name is not a keyword. Types are first-class objects in Unobtainabol and a type name is  just a name that happens to be bound to a type.

Because Unotbainabol has high dynamicness, variables do not have to be statically typed. Type declarations are optional, but the declaration of the name is required. Every name has to have a declaration that shows what its scope and lifetime is. Also, declarations can have initialization, so the above example could be rewritten as
do while x < y;
  var i;
  i  := 2*x;
  x := i;
od
or
do while x < y;
  var i := 2*x;
  x := i;
od
The loop is a highly flexible structure in Unobtainabol. Other examples are
do
  var z := f(x,y);
  for var i := 1 to z;
  p{x,i};
od
do
  var i := x+y;
  x := y-1;
  while x > y;
od

Saturday, April 20, 2013

Principles of Ultimate Programming Language Design

Here is a quick summary of the principles that should be followed in designing the ultimate unobtainable programming language. These principles may or may not apply to designing real programming languages.

Lets start with all of the mutually incompatible goals that the perfect language must satisfy.

Goals

  • Power. The language must have a full set of sophisticated and powerful features to let the programmer implement any sort of algorithm or paradigm that he has in mind. It also must provide all of the basic data structures such as expandable lists and sets and algorithms such as sorting.
  • Simplicity. The language must be easy to learn. This seems to contradict the requirement for power.
  • Standardization. The language should include a lot of standardized built-in functionality of the sort usually associated external libraries so that a programmer can come into a new project and already understand how things work. For example it ought to have standard support for dates and times, localization, internationalization, network services, 2D and 3D graphics, sound, user-interface widgets, database access, etc.
  • Flexibility. The language ought to support and encourage the creation of new libraries to improve on the the standard libraries. This seems to conflict with the purpose of standardization.
  • Predictability. The programmer ought to be able to look at a piece of code and know roughly what it does based on generic rules of the language. For example, the user ought to be able to guess that a function call does not modify its arguments or that all of the arguments to a function are evaluated before the function is called.
  • Extensibility. The language should allow the use of any parameter-passing mechanism and the creation of new syntax, new control structures, and pretty much new versions of anything that is already in the language. This seems to conflict with predictability.
  • High-levelness. The language must be highly abstract and automatically handle the low-level busywork such as storage management and managing a persistent (on-disk) store of language values.
  • Low-levelness. The language must allow the programmer get down to near assembly-language-like details on the operation of the machine and completely control storage management, disk usage, on-disk data structures, and the like. This seems to contradict the requirement for high-levelness.
  • Dynamicness. The language must allow a lot of decisions at runtime including creation of new functions at runtime, dynamic dispatch, late binding, reflection, and similar things.
  • Verifiability. The language must verify at compile time essential programmer assumptions such as the types of variables, the existence of function implementations and similar things in order to help ensure program correctness. This seems to be incompatible with dynamicness.
  • Safety. The language must dynamically ensure that if the programmer make mistakes that would invalidate the program state like dereferencing NULL pointers or overflowing a number then a clear message is sent back to the programmer with an immediate abort. Otherwise these sorts of errors can be very hard to reproduce and to track down.
  • Efficiency. The language must allow the programmer to skip checks such as testing for NULL pointers, checking for arithmetic overflow and similar issues when he knows that the check is not necessary.This seems to be incompatible with safety.

So how do we satisfy all of those incompatible goals? With some basic principles of ambivalent design.

Principles

  • Power/Simplicity. The language should contain a lot of powerful features, but should be decomposable into smaller, simpler sublanguages that can be taught and used without reference to the full language. It should be possible to start with a small teaching sublanguage that just introduces basic control structures and variables, then gradually adds features such as widgets and graphics.
  • Standardization/Flexibility. There should be lots of powerful standard libraries, but they should all have decomposable, pluggable features than can be replaced in sections. This lets programmers replace minimal parts of standard libraries and allows the users of those libraries to use largely standard interfaces.
  • Predictability/Extensbility. Powerful extensiblity features should be allowed but those features should be syntactically distinct from standard language features. For example, it should be possible to define a special function that does not always evaluate its arguments but the call should not look like a normal function call.
  • High-levelness/Low-levelness. The language should be fundamentally high-level but it should allow special escapes into a lower-level sublanguage. For example, the normal number type would be an arbitrary precision decimal number, but the user could declare some variables to be a 64-bit machine integers, which would cause arithmetic on those variables to use machine instructions. There should also be ways for users to control their own storage management, on-disk structures and other low-level features, but these facilities would be part of the low-level sublanguage, not visible to most programmers.
  • Dynamicness/Verifiability. The language should be fully dynamic but allow declarations that are checked at compile time. This lets a programmer write a small quick program using dynamic logic and then gradually refine it for verification and performance by adding declarations and assertions.
  • Safety/Efficiency. The language should have no undefined behavior but should have a sophisticated system of assertions to allow the programmer to prove to the compiler that a particular construct is safe without checking. The language should aggressively use these assertions to optimize away checks. The language should also do some sophisticated reasoning of its own and should have a facility to let the user know what it knows about values at compile time. In addition, it should provide an optimization facility to tell the programmer which checks might potentially be eliminated if they could be proven unnecessary.

Friday, April 19, 2013

some terminology for discussing programming languages

Abstraction : an immutable value that can be be referenced but not changed. In applicative languages, all values are abstractions. In languages with objects (which includes slots), an expression that references an abstraction can change, but only by being modified to reference a different abstraction. The abstraction itself does not change. This is in contrast to an object which can change while remaining the very same object. Numbers and other mathematical objects are typical abstractions. Queues and other dynamic data structures are typical objects

Active Slot: a slot that can cause a state change when dereferenced or can cause a state change when updated other than simply changing the current value in the slot.

Aliasing: the situation when two or more variables denote the same slot

Denotation: The entity that a name or other expression represents. In languages with slots, denotation is not the same as reference. A variable denotes a slot but references the value in the slot

Declaration: A syntactic construct that introduces a new name into a namespace. In many languages declarations are also associated with a constraint on the type of the name

Definition: A declaration that also specifies the entity that the name denotes. A definition is a form of constraint; it constrains the name to be equal to its denotation

Class: a set of values and operations on those values. A class is a property of a value, not of a name or variable. A value can have multiple classes, but each value has a distinct base class

Constant: a name that denotes a fixed value --that is, not a slot. For a constant, the denotation and reference are the same

Constraint: a proposition that must be satisfied. A type declaration is the typical kind of constraint. Pre-conditions, post-conditions, and compiler-checked invariants are also constraints. Constraints can be used as a form of redundancy to help ensure program correctness and can also be used to optimize execution. In logic programming languages, the entire program is a constraint and the process of finding a way to satisfy the constraint is the execution of the program

Entity: a language thing that can be denoted by a name or other syntactic construct but may not be a value. Typical non-value entities include types, state changes and slots. In some languages, functions are non-value entities. Entities are divided into two main categories: abstractions and objects, depending on whether they are immutable or mutable.

Expression: a phrase that represents a value.

Object: a mutable entity –an entity that can change over the the course of a program while remaining the same entity. For example, if a set has operations to insert and delete from the set then it is an object. A set that is fixed and unchangeable would be an abstraction rather than an object

Phrase: a program fragment that forms a syntactic unit. For example in C, "x=3" is a phrase because it forms a complete syntactic unit but "x=" is not, because it is an incomplete fragment of syntax.

Reference: the value that an expression references. If an expression denotes a value then it also references that value. If it denotes a slot, then it references the object in the slot. If it denotes some other entity that is not a value, then it has no reference. Typically operations and function calls act on the references of their operands or parameters

Slot: a special object that can be updated to take on different values dynamically. An expression that denotes a slot is said to reference the value in the slot. Slots are entities but not values

State Change: A change in the world represented by the program. Specifically, a change in one or more objects of the program because only objects can change. Machine level changes such as assignment of intermediate results to registers or the advance of the program counter do not count as state changes within the program.

Statement: a syntactic unit that represents a state change.

Type: a set of values that an expression can reference. A type is associated with expressions, not values. In other words, there is no such thing as the type of a value. Since a type is a set of values, a value can have many types. Values have a base class instead. A type can include all of the values of a given class, or of a set of classes or only some of the values of a class

Value: an entity that can be the reference of an expression, passed as a parameter to function, and returned by a function. Slots cannot be values. The value of an expression is the value referenced by the expression

Variable: a name that can change its reference over the course of a program. Not all variables can be updated. For example, the formal parameters of a function are variables even if they cannot be changed during a given call. They are variables because they take on different values for different calls

Thursday, April 18, 2013

a note on the unobtainable

I think my love for formalism started with 7th-grade algebra. I had never cared for mathematics before that. To me, mathematics was flash cards and long division and frustrating story problems that I never really got. But after the teach showed a few examples of solving simple algebraic equations I had an "AH HA!" moment. It was the first time that I had a glimmer of the power of formalism.

When I say "formalism", I"m talking about the use of symbols to represent abstract concepts, and the use of rules for manipulating the symbols that give you information about the abstract concepts represented. This is a remarkable thing, is it not? That we can take such an abstract thing as a number or a set --a thing that exists outside of time and space, beyond nature, only accessible by the human mind, and represent this magical thing with complex stains on a surface, then follow a sort of game of re-drawing those stains according to a set of rules, and at the end we know something about the abstract objects that we didn't know before.

Isn't that a marvelous thing? Isn't it something that would be completely unexpected and almost magical to a person who had not grown up with formalism and had it introduced to him later in life? Formalism is, in fact, one of the greatest tools of mankind, right up there with things like the wheel and fire and money. It is one of the fundamental discoveries that made civilization possible.

Programming languages are a kind of formalism that is intended to be manipulated by a programmable device rather than by a human. My love for programming languages began with CSc 227. I think the title of the course was something like Principles of Programming and it was taught in Pascal. I took the course, not because I wanted to learn any principles but because I wanted to learn Pascal which was supposedly the next big language.

Before that class I had programmed in Fortran and BASIC. I enjoyed the power of being able to control the computer, but neither language was particularly interesting. But Pascal was an eye-opener. I learned for the first time that a programming language could be elegant and that programs could be beautiful. Of course Pascal, lovely as it was, was imperfect.

In later semesters I was introduced to many other languages, all of which had interesting features but none of which were perfect. There was C with it's primitive control of machine instructions, Lisp with it's simple unified model of data and programs, APL with it's powerful set of array-processing primitives, SNOBOL with it's intricate string processing capabilities, COBOL with its support for persistent data stores, Algol --a more powerful Pascal-lke language.

What I never found was Unobtainabol, the perfect programming language (see Unobtanium for the inspiration of the name). I spent years in graduate school studying many more programming languages: Icon, FP, Ada, SL5, Smalltalk, C++, Scheme, Prolog, and many others. I tried to analyze what made languages good, and when I found that task impossible, I found a way to analyze simply what makes languages different (more on that in a future post).

I spent years studying, designing and inplementing programming language features and obsessing on my unobtainable goal of the perfect universal formalism --not only a programming language, but a language of mathematics and logic as well. Now that I have a real job, I still think about programming languages and other formalisms and how to improve them, so I decided to start a blog about it.