Saturday, August 11, 2018

What is a PICK MultiValue Database?

Why I'm Blogging This

In some upcoming posts, I'm going to be talking about how we implemented an ANSI/92 compliant SQL engine and ODBC driver for this, and some of the interesting challenges we encountered. In some cases, I'll be talking about the relative merits of these systems vs. relational databases.  They all have their strengths.

In order to do this, I thought it would be good to have a bit of a primer about how Pick data is stored and processed. I'll probably reference this little blog post quite a bit.

It's Still Here

The first thing I want to say is that systems running on Pick based multivalue are ubiquitous and they are EVERYWHERE! In some cases they are legacy solutions that have been in place and working for ever.  Probably every fortune 500 company has at least one system, running in some department, that runs on it.  It's also still being sold as part of applications around the world.

Examples of industries and solutions that continue to run and/or sell best-of-breed solutions running on PICK include (but are not limited to) Automotive Retail, Libraries, Banking, Healthcare, Insurance, Manufacturing, Distribution, EMS, Hospitality, Municipal, Government, and many more.

Ignore them at your peril!

Where Did it Come From?

In about 1965, the US department of defense awarded a contract to TRW to create a system to manage Cheyenne helicopter parts.  Two engineers, Don Nelson and Dick Pick were assigned to the task, and started work on an IBM System/360 computer. They named their system GIRLS, which was an acronym for Government Inventory Relational Language System.

This was before commodity hardware was a thing. Somehow they got the idea to create a system based on a virtual machine that they could implement on any hardware.  There was some really good architecture and design that went into this system.

How Does PICK Data Work?


The database was a very simple concept.  Whatever document you needed to represent, you would have a data structure that contained all the information in one place.  With a single read, a programmer could pull back all the key data that was needed for an invoice.  In real practice, your invoice would reference a customer by an id, and the customer name would remain in the customer master file. Similar things would happen with part numbers and the part names.  The invoice would include both the header information and the detail lines for the parts being ordered.

The data is stored in a hashed file structure and each record is stored as a sparse string with delimiters separating fields, values and subvalues.  Let's look at an example.

Still working with the invoice concept, lets consider a manufacturing organization that sells three key products, widgets, gidgets and gadgets.  The invoice file has an invoice date, customer id, then it has 4 fields that contain multiple values (multivalues) in them.  These are product id, options, quantity and price.  When you display an invoice it might look like this:

Invoice 0001 Date 11 Aug 2018 Customer ABC - ABC Company

Product Options Quantity   Price............ Extended...
WIDGET  GOLD           2            $115.95    $231.90
GIDGET  RED            1            $220.75    $220.75
        SMALL
========================================================
Subtotal:                                      $452.65
Taxes
Totals

When looking at the document, you can see that you have header data that occurs just once per document, including the invoice number, date, and customer information. You would look up the customer's name from the customer master file.  You also have computed or calculated fields, like extended costs, subtotals, taxes and final totals.

The interesting thing to note is the invoice lines data. In a real invoice all the lines, an set of arbitrary size, is part and parcel of, and contained in, the invoice document.  For a single invoice you could have an arbitrary number of products being purchased. for each you would have a quantity and a price.  In our case, we further complicated it by allowing different products to have different options. You might choose a colour and a size as with the gidget we had on the second invoice line.  At this point we not only have multiple values, but one of the values (the OPTIONS value for the GIDGET line) has two subvalues, one for the colour and one for the size.

Don and Dick came up with a database structure that allowed that.  Your fields were delimited by field or attribute mark delimiters. Your multiple values were delimited by value mark delimiters and your subvalues were delimited by subvalue mark delimiters. The data for the above INVOICE file might look like this:


Note that the blocks are the field delimiters. the superscripted '2' is a value mark, and the superscripted 'n' is a subvalue mark. These are high-ASCII delimiters. Going left to right, we have the first field, the item-id (similar to a primary key) which is the invoice number '00001'. Then you have a date. This is represented as number of days since Dec 31, 1967.  Next we have the customer id 'ABC' for ABC Company. Then we get a field that has two values. This field and all the subsequent fields have two values, the first one is line 1 of the detail lines and the second is line 2 of the detail lines. This first multivalued field has the product ids of the products being bought.The next one has the options selected for each line. You can see that for detail line 1 there is only one option, but for the 2nd detail line (the GIDGET), we have two options: The colour RED and the size SMALL. Then we have the quantities of each of these and the unit price for each line. The price has an implied 2 decimals (it is storing the value as cents, not decimal dollars).

For a programmer, the invoice number gave you the whole document. You might need to do ancilliary lookups in the CUSTOMER, PRODUCTS and OPTIONS tables, but a single read and a single write gets you your data and saves it to the database.

In a relational database, the INVOICE file would need to be a minimum of 3 tables. The record represented above would take a minimum of 6 disk reads to get the same data. As you increase the number of detail lines and the number of product options, the complexity grows rapidly.  A programmer in this case needs to retrieve the data from the database and make sure he gets it all. Then he needs to properly relate the data sets in his program. The complexity is huge compared with the multivalued approach. 

So, in relational terms, you would probably have these tables:  INVOICE_HEADER, INVOICE_LINES, INVOICE_LINES, INVOICE_LINES_OPTIONS.

Think of how this would work if you stored your physical invoices like this. You'd go to the filing cabinet with header information and look up the header part of the invoice. Then you'd go to another filing cabinet where you'd have to find a separate page for each invoice line. You'd use the invoice number and line number. You'd keep looking until you couldn't find another line number for that invoice. Then you'd go to yet another filing cabinet, and for each invoice line for that invoice, you'd look for one or more pages that had information for the options (possibly multiple) for those invoice lines. I've seen invoices that had hundreds of lines in them. Can you imagine then having to organize and manage those hundreds of pages?  Instead of one invoice that was maybe 3 pages long?

There is also a performance component to this.  I remember working with a customer who had moved from a mainframe to a minicomputer platform, and had determined that they could not afford the disk head movement of a relational database.  They were a big customer of our ODBC technology as we enabled them to have a relation view of the data and use it with tools that required this, while still giving them the reduced disk head movement for their core application.

How Did You Access That Data?

Pick had a concept called a dictionary. You would have a file that contained field definition records associated with every file. It was completely optional, and once the BASIC language came out for Pick, it didn't enforce anything in it. In fact, you could have two dictionaries that defined the same attribute completely differently. Generally, one would be wrong and the other right, but I've seen where a single file has multiple different record types, based on a prefix in the key. Which dictionary you wanted to use depended on the record key!

You could create a dictionary definition item that actually used a value from the current record to read a value from another file. This was called a translate, and was very powerful.  For the example index above, you would take the customer id (let's say it's attribute 2) and use it to pull attribute 1 (customer name) from the CUSTOMERS file. That translate correlative would look like this:

001 A
002 2
003 CUSTOMER NAME
...
007 TCUSTOMERS;X;1;1
...

Similarly, you could use this to pull information from a PRODUCTS file.

When these systems were first commercially sold, salespeople would walk a machine in from a panel van (they were the size of a fridge, so this was quite a feat.)  Then they'd use it to create a couple of tables, use the editor to create some dictionary definitions. Create a script program using PROC and BATCH to enter data. Then they'd use a LIST command to print out the data.

So the command "LIST INVOICES" would be automatically expanded out to:
LIST INVOICES INVOICE.DATE CUSTOMER.ID PRODUCT.ID OPTIONS QUANTITY PRICE

which would give us this output:


It was not uncommon for this to be so impressive for the business users watching the demo that a cheque would be written and the computer left there. Then they had to get programmers in and write a system... but it was also not uncommon for a business person to dictate what the system had to do and create his own reports.  Compared with the options available at that time, this was a huge step forward, and many commercial systems that are still in use today had their start with a business owner looking over a programmer's shoulders. This was the first instance of agile and pair programming!

Later Add-ons

The first systems were written using a combination of PROC (a scripting language), BATCH (which was horrible, and impossible to make pretty - totally character based), and when you got stuck, Assembler.

Later, a BASIC implementation was added with built-in support for the multivalued database, and that allowed you to use extended features of an ASCII terminal.  These databases still exist and go by a number of names:


  • D3
  • Universe
  • Unidata
  • mvBase
  • mvEnterprise
  • jBase
  • OpenInsight
  • QM and OpenQM
  • RealityX
  • And others

Relational Value and Pushback

An interesting thing to note is that many applications are being written today to use what are called NoSQL databases. The rationale for these databases sounds like an ad for Pick.  While relational databases have their uses, there are clearly applications where they are just not the right solution.  

That said, a huge amount of investment has been made by companies like Oracle, IBM, Microsoft, Cognos, Business Objects and others into enhancements and tools that leverage relational databases. This has not been the case with Pick, so there are many applications where it is imperative that the regardless where your data originates, you need to get it into relational in order to leverage these tools and technologies.

The industry is clearly seeing a divide form between SQL and NoSQL databases, yet there is also growing clarity about when you should use each.  What's not as clear is that there is a NoSQL option that has already existed for a long time and that is widely in use, that's another option.

Sunday, July 1, 2018

ODBC for PICK - Why You Shouldn't

Fundamental Mismatches

There were a number of very key, fundamental mismatches between PICK databases, and the ODBC model.  Enough in number and severity that many would say it's impossible, or "don't even try", but the group of pioneers that assailed this opportunity were not deterred by a few minor challenges.  Here are some of the things that we had to address.

Two Main Parts of ODBC

ODBC consisted of two main components. One was a SQL syntax, the other was an implementation of a Call Level Interface (CLI), which is much the same as an Application Programming Interface (API).  As an implementer of a driver, you could declare yourself as supporting one of three levels of each.  In our case we implemented the middle level of both.

Structured Query Language (SQL)

This was the biggest one, and I'll address it better in a separate post, but will touch on it here.  ODBC assumed SQL. While there were ways to bypass that, most tools that worked with ODBC would assume a certain level of SQL, and if you reported that you didn't support it, they'd just give up. So, if you wanted to support ODBC, you had no choice. You had to support SQL.  At the time that ODBC was released, while it was gaining popularity rapidly, both ISAM and Multivalued databases still outnumbered SQL databases.

SQL demanded certain norms, including that it was strongly typed and that it abhorred multivalues.  Multivalue databases like PICK and MUMPS (which was the other popular multivalued system, especially in health care) were, as the name implies, designed to work with multivalues. Again, I'll cover this in more detail elsewhere.  In short, the data models supported by each were very widely different.

Strong vs. Weak Data Typing

In PICK, everything is a string.  If anyone is familiar with XML, there are some definite similarities.  In XML you can have numbers (but they are simply strings containing numeric characters), and you can even have a schema that declares a certain XML element to be numeric, but in the end, if you don't enforce the schema in code, you can put what you want in that XML element. 

PICK had similar structures. In PICK, much like an XML schema, you had dictionaries. It was quite common over time, for PICK dictionaries to collect garbage. You'd have several conflicting dictionaries pointing at the same attribute (the PICK name for a field), and only one of them was really correct, or perhaps none of them were correct!  And it was not uncommon for a file's dictionary level to be empty.  Dictionaries were not enforced, but were useful for doing LIST, SORT, SELECT or SSELECT commands.  LIST and SORT were for creating user reports and SELECT or SSELECT were for activating a list of unique primary keys (called item-ids in PICK terminology.)

The CLI portion of ODBC, meanwhile, was designed for programming languages designed to produce machine code, and perform machine-level calls (hence "Call Level Interface").  This included Assembler, C, C++, and COBOL among other languages.  These languages were all strongly typed.  You would declare a variable as being a string (null terminated - no 16-bit Unicode in the early days - that came later) or an integer. The CLI had its own data types and would even declare the size of the storage for an integer.  The CLI had details of the sizing and precision of each data type, which you had to map to your machine's (or compiler's) data types.  This guaranteed that regardless whether your machine called a 32-bit integer a small int or a long int, you knew how big the data coming back from, or going into, the CLI had to be, and therefore could allocate enough space for it.  This allowed for interoperability between machine types, which was a key requirement for the standard.

Multi-threaded vs. Single User

PICK was a legacy system, originally built on the premise that a user was connecting to an expensive user license through a single, predefined serial port that ran through the building right to their terminal (or PC with a terminal emulator, like PK Harmony or Wintegrate.)  There was no multi-tasking and definitely no multi-threading in that system.

While ODBC didn't require you to support multi-threading, most applications using it expected to open multiple connections to do their work. Now, if you had two serial ports, you could open two connections, but this cost a lot, and required you to run another serial cable to your PC.  Even when we went to networked connections, the license cost for an extra PICK user was prohibitive, so this was another issue we needed to resolve.

Authentication and Authorization

In a pre-public-internet, pre-malicious-hacker age, we were very much aware of the security issues that we were likely to raise, and took these very seriously.  Although it was only a reality for a small number of users, network connectivity was already a reality for some, and we knew it was only a matter of time before it became the norm. We already had people dialing up systems over the telephone network, and I had seen people using dial-up, Prime-Net and then Internet (pre-commercial), and get connected to the wrong Prime Information system. This system was a NORAD system and had no password on the SYSPROG account (the PICK equivalent of root!) We logged off very quickly!

PICK's authentication and authorization model at that time was very weak and not designed for a broad network-connected world.

Performance

We were very much aware that we were likely to run into interesting performance problems for a number of reasons:

Data Architecture Mismatch

We were forcing round pegs into square holes. There were fundamental differences in how SQL and Multivalue databases architected their data and trying fool PICK into thinking it was SQL was likely to present some interesting challenges for performance.

Data transfer speed

When we started this, most PICK systems still used Serial I/O for their connectivity. There were solutions from Netware that allowed you to use the network, but the last bit of the connection was through a serial port concentrator, so you were still limited to the maximum speed of the serial I/O.  This was exacerbated by the fact that most PICK I/O channels designed for terminals would fall over if you burst data into them too quickly.

Challenge Accepted!

I've never found a challenge like this to be discouraging, but rather invigorating, and I've been blessed to work with many people who share that enthusiasm for solving difficult problems in elegant ways, so we got to work addressing the issues.  Challenge Accepted!

Friday, June 22, 2018

ODBC and SQL

Data Driver Difficulties

Round about 1990, my brother's friend John had a company called Paradigm that was writing drivers for Microsoft.  Structured Query Language (SQL) was becoming very popular, with Oracle the clear leader.  There was a problem, however.  It seems that every time you came up with a new application that needed to access data, you had to write a new driver for every database you wanted it to access. If you had, say, 5 applications that needed access to data and 5 databases that you wanted them to access, you had to write 25 drivers.  If the numbers became 10 and 10, you needed 100 drivers. This simply would not scale!

SQL Access Group (SAG) Call Level Interface (CLI) - Alias ODBC

Microsoft was not alone in struggling with this.In 1989 the X-Open Open Access Group had formed a specialized standards body called the SQL Access Group, with the initial members being Oracle Corporation, Informix, Ingres, DEC, Tandem, Sun and HP.  Microsoft joined round about 1990.

This group came up with a call level interface (CLI) that they defined for SQL databases.  It was intended to allow programming languages to access a single driver written a database vendor.  With this approach, if you had the 5 apps and 5 DBs listed above, you would need 5 drivers, and your apps would all need to have an interface provided that would talk to the drivers. 5 drivers instead of 25. 10 drivers instead of 100.

The SQL Access Group finally came up with a name for their standard. They called it Open DataBase Connectivity, or ODBC.  In addition to the CLI, this interface expected you to support some level of SQL. There was Minimum, Core and Extended. We implemented Core.

John's company was contracted to write drivers for all of Microsoft's supported ISAM databases, including comma-separated text files, DBase, Excel, and others.  This meant that they had to write a full SQL engine for these, as comma-separated text files had no SQL processing capability. John was working with a gentleman named Jim, who took on the task and did an excellent job.

In an interestingly twist, when a company called Fincentric wanted to convert their application development environment from using ISAM to SQL Server, Jim also wrote the inverse, taking ISAM commands and converting them into efficient SQL statements.

ODBC for PICK

In 1991, before ODBC even had a name, a number of us met at the India Gate restaurant on Robson street downtown, and discussed the idea of writing a SQL engine and driver for PICK. (We actually recorded notes on a napkin!)  Because of my involvement with Synex and PK Harmony, I was the communication expert.  John was the C/C++ expert with Jim as a very helpful resource on writing SQL engines for non-SQL environments, and my brother and I were the PICK experts.

We decided that my brother Antoon would take on a project working with SQL Server for Vancouver General Hospital (VGH) in order to better understand SQL, so we could determine the feasibility of what we were contemplating.

Laboratory Information Management System (LIMS)

The project was code-named LIMS. The funny thing was that Paradigm was assigned a phone number by VGH that had formerly been for some other department there.  Occasionally, the programmers would get a call from someone who didn't realize the number had been reassigned. They'd answer it with LIMS and there'd be a long pause at the other end, while the caller tried to figure out what department at VGH was called "Limbs"!

As a result of his work on this project, Antoon came up with an initial architecture and design for an ANSI 92 compliant SQL engine prototype, which he began coding in PICK BASIC.

It didn't take too long to get to where "SELECT * FROM SALES" returned a meaningful result set. That was the easy part.

ODBC vs. IDAPI - Oddball vs. Diaper

Borland created their own interface called the Integrated Database Application Programming Interface (IDAPI), which never took off beyond their own databases.  If you ran Microsoft Word's spell checker on IDAPI, you got "diaper". If you ran it against ODBC you got "oddball".

Microsoft ODBC Driver Developer Kit

We were one of the first to get a copy of Microsoft's ODBC Driver Developer Kit. In order to boost productivity, John took their API and re-wrote the header files to support C++, but expose the interfaces as C. I began writing the driver, and developed a transfer protocol that would work with Serial I/O, which was all that most PICK systems would support at that time. I used my experience with Synex, but based on a variation of the Motorola IXO transfer protocol that would work with serial I/O limitation of most PICK systems.

Microsoft had 3 levels of conformance to their API. Core, Level 1 or Level 2. We implemented Level 1 conformance.

Microsoft Releases ODBC 1.0 in 1992

Finally, in 1992, Microsoft released ODBC 1.0. Sort of... They implemented their driver manager so that it would only support their drivers for the first version.  We had been ready to release with them, but found out only weeks before they released that they had decided to implement this limitation.  I scrambled to write an Excel add-in that would allow us to use our driver to load Excel spreadsheets, so our beta customers would be able to use it.

We still managed to get a number of customers up and running with our beta version. And so our ODBC journey was launched.

We Failed the Bozo Test for ODBC

One last funny story for this post: It was an industry inside joke that if you could spell the letter of ODBC in the right order you probably knew what it was.  The unknowing would always refer to it as OBDC.

We put together a marketing brochure, and went through probably about 50 edits.  Somehow, on the very last edit, someone reversed the D and the B and we were so bug-eyed we missed it.  Our brochure went out with OBDC and we couldn't correct anyone who phoned in!

We failed our own Bozo Test!

Sunday, June 17, 2018

The Changing Face of Connectivity

In the Beginning...

One of my earliest customers was First City Trust. While they had started on Microdata, they had begun to move the applications to an IBM mainframe computer.  This was an interesting exercise, as they discovered that it was faster to move data back to the Microdata and get reports off of it from there, than to ask for them on the IBM. They had an army of people (an entire floor) that were focused on "fixing" data that was broken by the application running on the IBM, and getting anything new run up took forever.

On the other hand, they had less than half a dozen programmers, including myself, who were doing these reports and keeping the applications that were still on the Microdata running.  It used to be said that regardless of price "no one ever got fired for hiring IBM".  We'd just shake our heads...

In those days, you had two options for moving data over. One was to print a report and hire someone from the local office temp pool to re-enter it. The other was to write it to a 9-track tape in a compatible format, then read it on the other system.  I did quite a bit of work interpreting COBOL programs that were used to write to the tapes, reading them, converting from EBCDIC to ASCII, parsing them, and then writing them into the appropriate places in the Microdata. Then we'd write a report and usually have it out in under a day.

Let There be Serial Data Transfer Protocols

With the advent of PC Harmony and PK Harmony, we had a new way to move data around.  For the minicomputer world, you could hook up an IBM PC through an RS232 port (also called a Serial Input Output Port (Serial I/O Port, or just Serial Port for short).  These ports initially would allow you to push data through a modem, or directly cabled to a terminal or PC.  Directly connected, you could connect at 9600 baud. This equated in really rough numbers to 8 bits per 10 to 12 baud, so you were getting under 1K per second.  Over a modem, you'd be lucky to get 1200 baud.  A good typist was faster than a modem.

Over time, the quality of the serial ports, terminals and modems all improved, until you could finally get 56K baud modems and 19,200 baud terminals.

The IBM PC came with a serial port interrupt handler in the BIOS that could barely handle 1200 baud on a good day.  In order to get any more speed, terminal emulator writers had to override and replace that interrupt handler with their own that was optimized.

Serial Ports had another problem. They were unreliable. Even when you had hooked the interrupt handler, there were times when enough other interrupts took over the machine and you would drop characters. Or electrical interference, or line quality for a modem would result in corrupted data.  There were parity bits in the low-level protocol that would attempt to help you determine if corruption had happened, but they were typically just even or odd parity. If your corruption flipped two bits, the parity bit would look right, but the character would be bad.

So, in order to ensure that data got through, we developed a proprietary error correcting data transfer protocol.  We checksummed the whole data package, put a length prefix on it, and then checked the checksum at the other end.  We also ensured that every character was echoed before we sent the next one, and would time out and retry.

This was a lot of work, for a relatively slow process, but it beat having someone re-key the data, so it was used by many Business BASIC and Pick BASIC users for many years.

Then There was Netware

For a long time, Novell's Netware was king in the networking world.  I remember when they release a new feature. You could send data over the Netware network, and it would connect to a Serial Port concentrator, that was hooked up to your minicomputer.

That last bit of connectivity typically had the same 9600 baud or maybe 19,200 baud limitation, so it wasn't blindingly fast, and it also had the reliability issues that the minicomputer brought to the table, so you still needed the error correcting protocol, but it had one significant benefit.

Prior to this, if you had a factory, and you needed to put a new terminal in it, you had to run a Serial Cable all the way from your minicomputer to where the terminal needed to be.  It didn't matter how many cables you had already run all that way, you still had to do it.  And Serial I/O was only certified for a certain distance, so unless you paid for expensive repeaters, you were not going to have very good reliability.

Now, you could run a network out to the factory floor, and by dropping a line off the network to a new PC running terminal emulator software, you could add it with very little cabling and very little impact on reliability.  In order to use this network stuff, however, you had to implement a protocol called Netware Asynchronous Communication Services (NACS).

We had a few customers who used this interface, but networking was still fairly new, and required deep pockets to implement. Most Pick users didn't have those deep pockets.

An Explosion of Networks

With Novell initially leading the pack, networking continued to advance. When implementing networks in the early days, you would look at ARCNET, Token Ring, Ethernet, and then you would have to consider topologies. You could choose between Star, Bus, Ring or Tree topology.  While all this was going on, the early internet was being developed.  It used Ethernet, and used a Mesh topology.

Nowadays, the internet uses a combination of Ethernet, Wireless (WiFi), and cellular data technologies. The latter two are gradually outstripping the connected Ethernet for end-user devices.

In the mid-1990s at Liberty, we hired a student called Pardeep, to assist us with support. In speaking with him, he suggested we consider using Winsock libraries to write a TCP/IP socket driver.  Many of the minicomputers we were working with supported TCP/IP based Telnet services, and since he was familiar with the Berkeley Sockets, which Winsock was based on, we asked him to write us a socket driver. Our LWinSock.dll dynamic link library is used to this date at a number of companies including two fortune 500 companies, in our ODBC driver.  We also ported the same driver to JDBC, and used a variation of the code in our OLEDB driver and managed provider.  That little socket driver has seen a lot of use over the years! The beauty of this was that we were able to completely drop all the error correcting protocol and simply acknowledge receipt back to the sender.  We could also send large blocks and it would generally be handled provided we didn't overrun the buffers at the target end.

Was That a Bagle or a Nagle?

One thing that we noticed after a while with our socket driver, was that it would perform very slowly when smaller packets were being sent to the server from the PC. I remember doing careful analysis and concluding that sometime between us requesting a send and the server receiving it, something was introducing a precisely 200 millisecond delay.  It took a lot of digging to unearth the culprit, but we finally did.

When they first introduced Telnet protocol, the implementers discovered that if you had a lot of Telnet users typing characters, by computer terms, there was a large gap between each character, so it would send a character at a time in a packet. The overhead in a packet of 1 byte was huge and wasteful, so a gentlemen with the last name of "Nagle" came up with a strategy. Simply put, as the human typed, the client end of the Telnet program would gather up typed characters and not send them until either a certain number of characters had been gathered, or 200 milliseconds had elapsed since the first character was typed. This was a bit like holding the elevator door and poking your head out to see if anyone was coming before letting the elevator go.

This was called the Nagle algorithm and turned out to be the culprit.

It seems they assumed, if you were opening up a socket to a Telnet server that you would want the Nagle algorithm turned on. We turned it off and the small packets flew fast and furious!  Lesson learned.

Telus PureFibre ™

Today I have Telus PureFibre, and get well over 150 megabits per second. That's roughly 150,000,000 baud!  I can connect wirelessly or with a physical cable, and generally, the other end is the limiting factor for speed.  Reliable data transfer is pretty much a given. If your wireless connection goes down, you will get an error and may have to restart your transfer, but in general, if your network stays up, the data will get there, reliably and fast.

Hotels often provide internet connectivity, but then have hundreds of guests sharing the same point of presence, and it's generally fat enough for a family of 5 with 2 watching Netflix. It doesn't handle hundreds of users at once.  That said, if you can find a decent Starbucks or other point of presence, you can generally get good, fast, connectivity today.

I know that new protocols and new technologies are in the works.  I can't wait to see what the future holds!

Sunday, May 27, 2018

Microsoft's Office Productivity Software

An Unexpected Surprise

Some time after we released PK Harmony at Synex, several of the people I had known there left, and went to work for Microsoft.

One day, I got an unexpected package in the mail from Microsoft. It was a pretty big box.  In it was software diskettes and user guides for two products. One was Microsoft Excel version 3.0 and the other was Microsoft Word 1.2.

They both came with hard cover user guides that were about the size of a standard letter sized paper sheet.  They had complete descriptions of all the commands and options in the books.

Vastly Improved Word Processing


Prior to getting these programs, I had Lotus 1-2-3 for spreadsheet work, and when I needed to create documentation, I used a primitive program called RUNOFF which had been ported to Microdata.

At Synex, we had people doing technical writing using products like Pagemaker and Ventura Publisher, so I was familiar with WYSIWYG editors, but they were expensive and I didn't have one myself. That all changed when I got MS Word.  I got some great pointers from our technical writers on how to leverage styles, which I use to this day.

In high school I had taken a typing course, largely to fill in a gap, but because I thought it would be cool. The result was that I was one of the few programmers who actually knew how to type (most did chop-sticks typing - two index fingers) and as a result I could type very quickly.

Microsoft Office Suite

In a few short years, Microsoft bundled Word, Excel, PowerPoint and for the professional version, Access, all in a single bundle called Microsoft Office.  Microsoft continued to send me free versions to work with, including a standalone Word 6.0 version that could install on DOS or OS/2 Presentation Manager. (You didn't actually need Windows to run early versions of MS Word.)

Because of PK Harmony, and the need to support Mail-merge and other automation features, I began to become quite expert in these desktop productivity products, like Lotus 1-2-3, WordPerfect and others.  We did a lot of macro programming to automate loading data into these programs from people's Business BASIC and PICK systems.

I could write technical documentation and get it 75% of the way there. The technical writers did the last 25% of the formatting and cleanup, giving it that professional shine.

Object Linking and Embedding and Component Object Model

As Microsoft developed their operating system platform, a new approach to integrating applications began to appear.  Through Object Linking and Embedding (OLE) and Component Object Model (COM) an application could expose its functionality so that another program could easily integrate with it.

Just a few years ago, I used the Microsoft Excel COM automation libraries to write a C# program to extract insurance rate data from an Excel spreadsheet that the executive had used to model their next period's rates, and push it into their multivalued mvBase database that ran their insurance line of business. The program I wrote ran through and validated that the spreadsheet had the right tabs, that the tabs had the right columns, and that the columns had the right data and that the data relationships made sense.  If anything was wrong we reported a highly descriptive and detailed error and stopped. If all was good, we used our own libraries, licensed by the user, to construct multivalued records and write them to the appropriate staging file (all done in C# - we had a library of multivalue dynamic array handling methods written in C#). The customer would check that file and validate it before replacing the live file with the data we uploaded.

Before OLE and COM, we would have required access to the source code for Excel, or would have had to export the data using a macro, or most likely would have hired someone to data enter it all.

Microsoft Office Today

Today, we have Office 365. While you get access to fat-client programs that you install and run on your device, you can store your data in the cloud, and use slightly limited web versions of the software from anywhere.  What's more, my subscription allows me to have Word and Excel running on my laptop, my Mac, my iPad and my iPhone, and they can access data on OneDrive, so they can all access the same information. I can switch where I'm working and continue where I left off!

As you may have guessed, I'm a great fan of the Microsoft Office suite of products.

Saturday, May 19, 2018

Microsoft MVP Reconnect

A New Community from Microsoft

The other day, my colleague, who is a current Microsoft MVP, told me about a program that Microsoft was putting together to bring past MVPs together in a community. The program is called MVP Reconnect.

I was awarded Microsoft MVP status for two different areas: The first area was ODBC, the second was for Win32.  Both of these are far enough back that Microsoft doesn't have records, so I had to pull up and email them scans of the various letters I had received in 1995 and in 2000 for my MVP awards.  About a week ago (May 2018) I was accepted in the MVP Reconnect community.

® Microsoft is a registered trademark of Microsoft Corporation in the United States and/or other countries.

MVP for ODBC Contributions

Back in 1995 when the Microsoft ODBC forum was hosted on the Compuserve network, I found that I answered more questions than I asked when I was on there. As a person who developed and deployed ODBC based solutions, and write ODBC drivers and back-end server software to serve the SQL syntax that was key to ODBC, I found that I had a unique understanding of how the middleware worked, so I was able to help many people with answers, regardless what driver or applications they were using.

Back in the day, some of the most prolific members of that forum included Microsoft employees: Murali Venkatrao, and Mike Pizzo, and non-Microsoft people: Lee Fesperman, Ronald Laeremans, Charles, McDevitt, Rob Macdonald, Mark Edwards and Dale Hunscher.  And many more whose names don't come to me now.  Between us we helped a lot of people in the early days of ODBC.

Microsoft release ODBC 1.0 in 1992. Unfortunately, the first ODBC-compatible versions of their Office suite really didn't play well with anyone else's ODBC drivers. Their applications only supported their own drivers. It was the next version of Office that finally provided real ODBC connectivity.

By 1995, my contributions were recognized, and I became an MVP (the program started in 1993, I believe.)  This award continued until they moved all Microsoft forums from Compuserve to MSN in 1998.  Unfortunately, I was running on Windows NT, and you needed Windows 95 to access MSN, which had a proprietary front-end application, so I got knocked off the network. Even if I had had access, no one posted any questions to the MSN network.  I continued to respond on the Compuserve network and the ODBC forum owners gave me free access for several years beyond that because of my contributions.

MVP for Win32

Then in 2000, I was awarded MVP status again, because of contributions I was making to Win32 developer forums.  That was before .NET was released, and I was doing a lot of C++ development.

MVP Reconnect

With all the attendant perks, and the chance to reconnect with some community minded industry experts, this was a pleasant surprise! I'm looking forward to interacting with this community!

Saturday, April 28, 2018

Playing Games on Computers

My First Computer Game

My first computer game really wasn't my own, it was my brother's.  I had two brothers, 6 and 7 years older than me, and the younger of the two had a Tandy Radio Shack TRS80 computer. It used our TV as a monitor, had a built-in keyboard, a game controller, and a cassette interface to back up and restore data, and to play games from.

It had a space invaders game where you went around the galaxy shooting up alien ships, periodically returning to your space station to refuel.

When I finally got a chance to play it, at about 14 or 15 years old, I had a nefarious plan.  I shot up all the aliens except one, figured out where he was, then went back to the space station and refueled. As I left the space station I blew it up.  A message about a pending court martial scrolled across my screen, but it let me continue on. 

So I continued on to where the last alien ship was and blew it up.  My plan had worked. I controlled the galaxy as supreme ruler (basically the only one with a space ship with weapons!)

Character Based Games

When I finally got into computers myself, it was initially on mini-computers with ASCII terminals.  Many of the terminals when I first started could not handle lower-case characters, and don't even think about graphics, so you were limited in what you could do.

I remember one game we played on our Reality systems, I think it was called Hammurabi.  You were buying and selling on the open seas in the ancient world. There were money lenders, including Hammurabi, who'd charge ridiculous interest rates and would chase you down looking for payment periodically.  

My brother-in-law got to playing the game, and discovered that when the money-lenders offered to lend money, you could type a negative number. They would basically have borrowed from you, and continued to charge (themselves) usurious interest rates and the money in your account would grow dramatically, with compound interest!.  He got so rich, his wealth finally exceeding the numeric capability of the computer and it crashed the game!

Microsoft Flight Simulator

When I finally got a personal computer, one of the first games I got was Microsoft Flight Simulator. This was an amazing game.  

Apparently real pilots liked to use it, because it was so realistic in so many ways.  You could select the type of plane you wanted to fly, and my pilot friend explained that the game would simulate both the correct cockpit controls and other quirky characteristics of the plane.

I loved that game and enjoyed quite a few hours on it. I was extremely disappointed when I finally upgraded from my old IBM PC XT with it's 4.77 megahertz clock-speed to an IBM AT clone with a 16 megahertz clock speed, and discovered that the game based its timing, not on the clock, but on CPU instruction times.  At about 4 times faster, the game would get you up in the air and tunneled back into the ground so quickly you couldn't really control it!

Games with my Son

I wasn't really into games that much, but as my son got a bit older, I began to look for games we could play on the computer and enjoy some family time. It was the latter half of the 1990s and my computer had progressed and it was now capable of VGA graphics and Soundblaster sound, so we began to find some really fun shareware games.  While many of these would support game controllers, finding game controllers that were any good for a PC was a pursuit we gave up on. 

We had several Commander Keen games, including our favourite, Keen Dreams. Little commander keen is wandering about in his dreams in PJs and slippers and defeating slugs to get back to his bedroom.

Another was Wacky Wheels. We could actually both play it at the same time by taking different sides of the keyboard.  You were driving racing cars as hedge-hogs and shooting mini-hedge-hogs at the other driver.  It was a blast!

We also found a number of Duke Nukem versions, and really enjoyed them.

Similar to Flight Simulator, as PCs progressed, these games "broke", yet using the virtualization of the PC, some very smart people came up with a program called DOSBOX that mimics the behaviour of an old DOS based PC, with Soundblaster sound cards and VGA graphics.  I can still play the Commander Keen, Duke Nukem and Whacky Wheels games using DOSBOX, and occasionally do (my son does, too!)

A friend gave us a copy of a game called Arthur's Teacher Trouble, which was another cute game.  We discovered some hidden Easter eggs in that game and it was a great favourite for our entire family, including nieces and nephews, for a long time, until Windows upgrades broke it, too.

As a result of my second stint as a Microsoft MVP, I got a copy of Monster Truck Madness, and more fun with my son and nephews ensued. There was a level where, if you went off-road, you could crash into cows and push them around. They'd "moo" and in the background a farmer's voice could be heard saying things like "Hey! You kids! Get off my farm!"  Another level had a trailer park in the desert with outhouses. If you crashed into an outhouse, you'd hear people inside yelling at you to stop.  As you can imagine, my preteen son and nephews got some great giggles from this.

Riddle of Master Lu

One of the people it was my pleasure to work with was my friend Pardeep.  After working for us for a time, he went on to a company called Sanctuary Woods, and helped them create a game called "The Riddle of Master Lu". This was a first person player game where you were Robert Ripley, looking for some treasure.  He was very handsome and got chosen to play the part of a monk in the game.  It was really cool to have a game where one of my former colleagues was an actor and a developer!

My Favourite Computer Game of All Time

And so we come to my favourite game of all.  About 25 years ago, I discovered a game at a trade-show. The graphics were amazing and it looked like a really fun first-person exploration game. It was called Myst.  

My son and I both checked it out, and quickly realized that our current PC simply didn't have a powerful enough graphics card. Worse yet, our hardware was incapable of supporting any of the required graphics cards.  Undaunted, for the first time ever, I upgraded a PC for personal reasons, not business reasons.  I got a new PC that would support a required graphics card and had them install the card for me.  Then I bought Myst. And Uru.  And all the follow-up games that came out from Cyan in that series.  We loved it!

The graphics were stunning. The story-line was amazing. The game was challenging, interesting, and engaging, so that you felt it was really you in there.  The realism of the story line and how things progressed was very good. And as a parent I was quite pleased that it was pretty-well completely non-violent.

As operating systems matured, and the user interface layer was taken out of protected mode (to mitigate the impact of exploits to the graphics layer), the games were essentially broken. I understand that they were written to work with a limited set of specialized graphics cards. It became increasingly difficult to find computers that would run the games, and they fell out of the mainstream.  There was one attempt to do an online version of the games, that I don't think went very well.

The good news it that Cyan is now running a Kickstarter campaign to create a 25th Anniversary Myst collection.  Needless to say, I'm backing it and looking forward to getting my copy of the series this summer!

Friday, March 30, 2018

Glimpses Into the Future

Working With Smart People

I've mentioned in past blogs some of the people that I've had the unique pleasure and good fortune of working with.  Some of these people were visionary, out-of-the-box thinkers, and viewed technology as a springboard into new and exciting frontiers.

As I worked with these people, their way of thinking rubbed off on me. I think I was already a bit predisposed to thinking like them, but they certainly accelerated the thought processes.

Here are a few of the ways that they influenced my thinking.

Unix vs. Windows vs. PICK

I remember that there was a conference held in Canada shortly after Expo 86 (or possibly the same year?).  This conference included several key speakers, for an important debate.  The debate was a critical comparison of Unix, vs. Windows, vs PICK.

Most PICK systems still ran as full-fledged operating systems, not database shells hosted on another operating system, so giving them a voice, for the organizers, made perfect sense. It was the battle of the operating systems.

I want you to remember that at that time, SQL databases were very new. Most business applications used ISAM databases, a slightly smaller set used multivalue variations like PICK or MUMPS, with PICK being the front runner, and a very small number used SQL of some sort.

In excess of 90% of north american library systems were based on PICK.  In excess of 95% of automobile dealer management systems were either ADP on PICK or Reynolds and Reynolds on PICK.  Many manufacturing systems were PICK-based.  I can name numerous fortune 500 companies that still have PICK-based applications, and PICK at that time existed in a majority, if not all, of the fortune 500.  Prime Information, which was PICK implemented as a shell on PrimeOS, was huge in government and the military (the entire NORAD system ran on Prime), and there were many other examples.  As a result, ignoring PICK in the debate would have been seen as invalidating the debate.  The organizers were possibly unaware of the growing groundswell of PICK variants that were being developed as a shell on top of Unix.  Prime Information had forged the way, very early on, as they developed a shell on top of the somewhat Unix-like PrimeOS, but the majority of PICK applications were still running on platforms that provided all O/S services as well as the database, run-time, and BASIC compiler and interpreter services.

So the organizers asked someone from Microsoft, a Unix technology specialist (from AT&T, if I recall correctly), and Dick Pick, to speak.  I got to pick up the Microsoft speaker at the airport and take him to his hotel. On the drive over, I told him that I thought the debate was wrong-headed.  That, in my opinion, he would see the day when a Microsoft computer would be the user's workstation, that connected them to an application, in some cases running on some version of PICK, implemented as a shell on another operating system, probably Unix (Windows NT didn't exist yet.)  He was quite intrigued.

As an interesting note, my current company has an army of Windows users, who connect to a legacy application written in jBASIC (a PICK BASIC variant) running on a version of PICK called jBASE, hosted on Unix servers, with an underlying Oracle Database storing the multivalued, non-relational data, so the reality was even more complex, and more specialized that anyone had dreamed of back in the 1980s!

News and Shopping Online

Then there were the networks. I remember using networks like PrimeNet, the Internet, AOL and Compuserve.  In the day, Compuserve was the big one, especially for commercial use.  As I worked with them and saw the slow emergence of visual, video and voice capability, and playing with video editing technology, the potential began to become clear to me.

I remember telling my wife and family that one day she would get most of her news online, that she would buy products online, and be able to share information and communicate with her family, including in remote places, through an online experience.  She laughed at the time, but she isn't laughing now!  We buy things online, we get our news online. We have a Netflix subscription and watch interesting stuff on YouTube.  We buy things on Amazon and Etsy and my wife sells stuff on the Internet.

Interconnected World

I'll leave you with this. 

I remember when someone I knew from Synex, called Chris, went to Microsoft, and helped develop a concept called Object Linking and Embedding (OLE).  A few years later, watching the whole OLE and COM world expand, and seeing how web-based APIs were developing, I was working with a group of people who were developing a standard for database connectivity.

I've watched as the web was invented on the Internet. From Netscape Enterprise Server, Microsoft Internet Information Server, Apache Web Server, serving pages to web browsers, and then to pagers, mobile phones, and other remote devices, I've seen the power and flexibility of connected systems.

It was apparent to me that connectivity was going to be a huge and exciting area of work.  It was also clear that we were gradually working towards a world were applications would eventually be able to work with each other, even if neither one had anticipated the value of a combined experience.

Applications that tried to play hard-to-get would die a lonely and painful death. Applications and vendors who understood the value of and need for collaboration on standards, would be the winners.

We aren't there yet. Standards need to be developed, then they need to be refined so they actually work, then they need to be adhered to, even while they are extended to support the next big thing that comes along, and it all has to happen at light-speed.

I believe that a combination of Internet of Things (IoT), Artificial Intelligence (AI), Social Media, Crowdsourcing, 3D printing, Blockchain, and wearable technology will continue to drive new disruptors.  And then there will be the next few "new things" that will blow our minds!

These are exciting times to be involved in technology!

Monday, March 19, 2018

Funny Support Stories

Through the course of my years in software, we had many encounters with technical support issues or bugs that were funny, either at the time, or looking back at them later.  Here are a few of them.

Close The Door

In the early days of personal computers, it seemed that no matter how well you documented instructions for installation of your software, a significant number of very non-technical users needed you to hold their hands through installation.  I remember at Synex, one of the support technicians almost falling out of her chair with laughter. When she finally finished the call, she told us what had happened.

She had helped the customer find the floppy drive on their PC, and told them to take the floppy out of it's paper sleeve, then slide it in with the logo on top and the open part in first.  Then she told the customer, over the phone, close the door. This of course meant the floppy door.

The customer said "just a second" and she could hear his office door closing in the background.  She almost died!

Where's The Any Key?

This is one of the old cliches of tech support, but I can assure you that it happened more than once. It was not uncommon for PC software to tell you to do something, then "press any key to continue".  More than one customer called us in a panic as no key on their keyboard was labelled "Any".

Corrupted 9-Track Tapes

Back in the day, telephones didn't have ring-tones, they actually rang. They rang by oscillating a magnetic field that made a little hammer wack two sides of a little semi-spherical metal bell on the underside of the telephone.  On occasion, someone would place an office phone on top of a stack of 9-track tapes. The phone would ring, and the top tape would be very effectively erased. Depending on how effective the little magnet was, you might partially erase the next tape down, too.

Putting one of these phones on top of floppies had the same effect, as floppies, like tapes, were magnetic media.

Fridge Magnets and Floppy Drives Don't Mix

On a similar vein, if a user had a bad time reading a floppy drive and it appeared to have been erased, it was often helpful to ask where the floppy was before getting to the computer. On occasion someone would stick it to a fridge with a fridge magnet ("I've never lost the fridge!").  This always worked as they were very thin and the magnet would hold it to the fridge quite nicely.  Needless to say, the magnetic data was nicely erased in the process!

The Wrong Floppy Type

The old 5 1/4 inch floppies held limited data, were big and a bit fragile.  So with the IBM PS/2 computer (not Play Station/2, Personal System/2), IBM came out with a new floppy, the 3 1/2 inch floppy. It held considerably more data than the 5 1/4 inch floppies, and a sturdier hard plastic case (it wasn't really "floppy" anymore) and was actually better technology.

We had a customer that we sent PK Harmony to on 5 1/4 inch floppy. When he couldn't fit it into his 3 1/2 inch drive, he tried folding it before calling us!

We sent him the right media and made a point of asking every customer what type of floppy drive they had before sending them media.

Providing Technical Support to GOD

At Liberty, we had bi-weekly support meetings to review how our tech support was doing in terms of meeting our SLAs for customers.  We had acronyms for most of our customers, and one of them was a company called Guaranteed Overnight Delivery.  Their acronym was GOD. This made for very interesting support meeting agendas and minutes!  It seems our technical support department was very helpful to the Almighty!

Mouse in the Agitator

I mentioned this story in a previous blog post from a few years ago, but I'll repeat it here.

At Liberty, we had a customer who had a D3 PICK system, and they were doing research for a cure for AIDS.  They did a lot of blood testing and had all the necessary equipment for doing it.

From the time that ODBC was first developed, Microsoft added a query tool called MSQuery to their Excel spreadsheet product.  This query tool is still there, largely unmodified. It does its job quite nicely! Unfortunately, MSQuery developed a bug that caused problems on just some computers. I believe that bug is long resolved but at the time it drove us nuts!  

This tool used the multi-threaded apartment model.  Even though you could have it pulling data back on a thread that was distinct from the thread that was updating the UI, every time you wanted to notify the UI that some number of rows had been pulled back, so it could update the running counter, you would have to call into the UI thread.  This meant that you put a blocking call onto the message pump for the UI.  This was not a big deal usually, but MSQuery would run into a situation where the message pump would get stuck unless you provided some user input to it.  This was not just a problem for us, but for any ODBC driver, it seemed.

This is how the bug manifested itself.  When you were doing a query, the little ODBC "World" icon would spin. When this problem happened, the world would stop spinning and nothing would happen until you moved the mouse or pressed a key on the keyboard.  Then it would start up again for a couple minutes before stalling once again.  Those stalls would stop both the UI updates and the query from retrieving data.

Our customer had run into this, and had a brilliant solution. He took his mouse, put it in a specimen agitator, and left it shaking until the query finished.  There was a constant stream of mouse movements that kept the query happening!  That was absolutely brilliant, out-of-the-box thinking on the part of our customer!

And that's one of the best things about my work over the years, getting to work with so many brilliant, kind, funny, good people!

Friday, March 16, 2018

Copy Protection Backfires

The Problem

Although there were lots of other personal computers, the IBM PC was the one that really took off for business use, and quickly surpassed all others, including Apple.  As the IBM PC and clones began to gain significant traction, software developers were faced with a dilemma.  The world of Software was a bit of a wild west. Was software subject to patent law? The Patent and Trademark Office of the United States initially said "NO!"  Software was like a mathematical algorithm, they said, and as such was not subject to patent law.  Was it subject to copyright? If so, did changing the variable names, and moving some subroutines around constitute a unique work?  Most software was delivered on 5 1/4 inch floppy drives. The YouTube clip below gives a demonstration of these floppies.



From day one developers of software were paranoid about someone stealing their software and reselling it.  Anyone with 2 floppy drives could make a full copy of one floppy to another.  It was easy to print out labels that would look like the original software. You could then resell potentially hundreds of copies of someone's software.  You'd get all the profit but the original authors got nothing. Because you didn't have the cost of developing or supporting the software, you could undercut them.

Copylock Protection

One solution to this was a product called Copylock.  They came up with a way to write to a track on the floppy, with special hardware, in such a way the the drives could read the track, but when copying the track, it defied the normal formatting and the drives would not copy the track.

Details of how Copylock worked are available here:

One of the most successful PC software companies, one that used Copylock protection, was Lotus. Lotus 1-2-3 came on 5 1/4 inch floppies with Copylock protection.  It was not uncommon for users to encounter a problem with their installation and would need to re-install Lotus. Copylock kept a count of how many times you had installed it, and it would only allow 3 installations before telling you that you had run out.  You could uninstall Lotus from a computer and it would increment its counter, but if your hard drive packed it in, or your 8 year old had discovered the "del" command, you had lost an installation.

Prior to my joining them, Synex had sent a software product that they had developed to an American software publishing company, in a bid to convince them to resell it. The company asked to see the source code, ostensibly to see its quality, and Synex sent it to them. That company told them shortly thereafter that they weren't interested in the software. A month later that company released it under their own name.  Remember that this was the wild west. There was no precedent of anyone being successfully sued for copyright over software, so Synex didn't pursue it.

As a result, they were very paranoid about piracy.  For their PC Harmony products for  Business BASIC and Wang BASIC they enlisted Copylock to protect their software.

When I joined Synex as their development manager, I noticed that we had a significant number of support incidents around people running out of installs on their floppy disks. In addition to taking a lot of support time, we had the cost of mailing them another floppy disk. Then one day, one of our Lotus 1-2-3 disks ran out of installs. I phoned Lotus support, who sent me another floppy, but instead of being another 1-2-3 diskette, it was software to disable the copy protection. They had figured out that the cost of supporting Copylock was higher than the risk of someone making a copy of the disk!

Removing Protectionism

I tracked for a time how much of our software support was due to Copylock issues and we quickly decided that we would stop using Copylock.  Our support costs dropped and our sales stayed steady!

One thing we had learned was that our product was complex enough that most users would want support, at least for installing the host software, so there was no real value in using the Copylock protection. The complexity of our product coupled with our excellent customer support was protection enough!  Lesson learned!

Friday, March 2, 2018

I Got Hired by Synex Systems

After working with Synex as a contractor from DataSense / Escom, I switched jobs to Synex Systems to become their Development Manager for PK Harmony and the PC Harmony products.

Synex was an interesting company.  One of the key people at Synex was Chris Graham.  In addition to terminal emulators, they were writing add-ins for Lotus 1-2-3.  They had a couple of Compaq luggable computers.  Below is a video clip of someone demonstrating one.



Murray my boss told me of going on a trip. Back in the day the airlines didn't let you take a computer as carry-on, so he had to check them in. He was telling the person beside him, in the window seat, about his concern that his computers would be OK, when the guy looked out the window and told him "I think your computers just fell 5 feet off the conveyor onto the tarmac!"  When he got them back to the office, he re-seated the boards in them, turned them on. One needed a new monitor and both cases were wrecked, but both ran just fine. Those Compaqs were truly rugged!

You could also get them with a built-in acoustic coupler, so you could put the phone receiver into it and dial up another computer.

Back to add-ins for Lotus 1-2-3.  You could only write them in Assembler, which was a pretty slow way to program.  There was a special piece of hardware call an Antron Probe, that would mount over the CPU chip. It would track all instructions going through the CPU and could back-trace these for you.  Chris took one of the luggables home for the weekend and reverse-engineered every entry point into DOS that Lotus 1-2-3 called, creating back-traces in assembler.  He then figured out how to hook a C program in.  Productivity for Lotus add-ins was immediately improved!

Chris later went on to be Director of Interoperability for Microsoft, and was a key contributor to Windows 3.0. The Easter Egg that listed Bill Gates and all the other key contributors listed Chris along with the others.

I also recall seeing a complete IBM BIOS assembler listing when I was still consulting there.  The draw was irresistible!  I had to work there!

It didn't take long for me to get my first Mark Williams C compiler. I poured through the Kernighan and Ritchie book, and went through the PC Harmony C and Assembler code.  I was hooked!

This was a time when it was possible to have a grasp of what was happening from your software right down to the hardware, before computers turned into onions with many layers.  You could order a set of Intel developer manuals and they'd ship them to you for free.  The first manuals were about 3 or 4 volumes, but over time they've grown. The last hard-copy ones I ordered were about 6 volumes and today you get them electronically. They are 10 volumes, but since they are electronic, they'll let you download them as 4 massive volumes instead.  I've always felt that the ability to visualize what is happening at different layers of a multi-layered, even distributed, application, is key to being able to architect, design and troubleshoot it.

At one point, I got to use a Compaq luggable that we called the "brick".



In addition to DOS, it had this cool new graphical Microsoft program called Windows 286.  It would do CGA graphics on the monochrome amber monitor.  It could also dual-boot into IBM's equivalent to DOS called OS/2 which had its own graphical component called Presentation Manager.

It was a really cool time to be working with computer software, and I was having a blast!

Saturday, February 24, 2018

PK Harmony - Terminal Emulation for PICK, and more

The Idea


In another blog post I talked about how Synex Systems had taken what we did for Terry Winter and turned it into a terminal emulator that could capture data from Business BASIC and Wang BASIC systems.  The owners of that company decided to pursue creating a version that worked with PICK systems.  A deal was offered to me that included me getting my own IBM PC computer in return for doing this little project.  I accepted the deal with great excitement!

The initial concept was to copy what the PC Harmony product did. The idea was that the terminal emulator software would respond to a special escape sequence, that was not used by any terminal we knew of, to shift into data capture mode.  It would then screen-scrape the data into an ASCII file, and another tool would allow you to define a ruler on the columns and capture the data.

This had some limitations. If you had summary lines, they often didn't line up, and created problems.  Also, when data wrapped in a column, it created interesting problems.

PICK had a query language called English on Reality systems, and Access on PICK systems (it actually had a bunch of names, but those two were the most common.)  It started with "LIST {filename}..." or "SORT {filename}..." and then you specified dictionaries for the columns you wanted on your report.  Dictionaries ranged from simple references to a field, to complex computed values. There were no joins, but you could reach into another file, if you could find its key, and pull out fields.  It was a pretty rich language and was very commonly used.

The Solution

So, I wrote a PICK/BASIC program that mimicked the LIST/SORT commands, but was called PICK-PC, and instead of padding and wrapping data for a columnar format, it delimited it.  Because both the PC and the PICK system had problems with high-speed transfers, and because hardware and software interrupts on the PC could cause it to lose data, we had to create an error correcting data transfer protocol for serial I/O.

The first version didn't handle some of the more complex computed dictionaries, but you could use raw data and could use "translates" to pull data out of other files.  In a pinch, you could stage the data you wanted into another file and then PICK-PC the data over.  I worked with the PC developers to come up with an architecture and design that would be practical on the PICK side, but still leverage the power of the PC.

More Horsepower Needed

It didn't take too long to realize that I had a little problem. I did much of this development (to earn my new PC) at home. But at home I could either boot into my 5 MB PICK partition, or my 5 MB DOS partition.  It was impossible to test PICK pushing data at the DOS side of the PC, as only one could be active at a time.  I also quickly ran out of room on my 10MB drive, so the company gave me a second hard drive. A 20 MB Seagate drive.

After a while, the company bought two AT&T 6300 computers. These were clones that were equivalent the the IBM AT class machine, which was the next generation after the XT. The video below shows one of these booting up.



Now I had a PICK system and a DOS system that I could run at the same time.  Both computers were quite a bit faster than my XT class machine, too, so that was another benefit.

Additional Features

In addition, I designed and the team developed a number of additional features:

  • A scripting language that the PICK system could use to drive PC activity. You could do a number of things like change your current directory, create a directory, check for existence of a file, run a DOS command, program or batch file.  
  • A scripting language that you could run on the PC to drive activity on the PICK application.  
  • We allowed you to take data out of several formats and upload it into a file on your PICK system.
The full feature list of the production version is listed in the product brochure below:



My First Spectrum Show


We went to a Spectrum show in Las Vegas to show off our prototype and get market feedback.  That Spectrum show had a special room called "PC Labs", and we had a booth there.  The first time I was doing the demo, I showed a LIST statement displaying some data on a screen.  Then I said "Now here's something you haven't seen in PICK." and showed them the PICK-PC command.

It was completely unrehearsed, but that line became our "hook". Everyone for 3 or 4 booths away trotted over to see this "new thing they'd never seen on PICK"!

We pulled up the same data in Lotus 1-2-3.  We did another set of data and pushed into a Wordstar mail merge.

Trivia Note: Wordstar was the first WYSIWYG word processor with mail merge capability.

We left that show with pre-orders for the product, and a list of interested beta customers!

What I Learned


For me, there were several things I learned:
  • I discovered PC programming and was intrigued at the possibilities.
  • I discovered the incredible rush of working with a brilliant team of creative geniuses to come up with something of value. Software is so much like magic. You start with an idea in your head and wind up with something valuable that people are willing to pay money for!
  • I discovered the excitement of working with sales and marketing teams to get something I helped author go to market!
  • I learned more about serial I/O and data communications.
Finally, although we were far from accomplishing our vision for the host scripting we had developed, that vision was in line with a recent trend called Robotic Process Automation.  Interesting to see it coming to fruition some 30 years later!

Note: For a long time, the computer systems used by most libraries in North America was PICK based system sold by Dynix. Synex had an annual contract to sell Dynix PK Harmony licenses in bulk to allow data transfers to other systems.  Our customers also included the largest McDonald's franchise in North America (in Florida), the London Underground system, and other companies, including  fortune 500 companies, that were using PICK systems in all industries and in countries around the world.  This was great exposure for me.

Sunday, February 18, 2018

Data Transfer from COBOL to REALITY and Back

First City Trust

While working for DataSense, one of the more active customers was First City Trust.  DataSense had software that was built for Canadian Trust companies, and First City was a big customer of theirs. I worked with a number of other Trust companies but First City was the one I spent the most time with.

First City had migrated their core deposit taking software to an IBM mainframe, but continued to run a number of business units on the Microdata Reality system.  This included mortgage banking (where they administered loans) and leasing. There may have been more but those two come to mind.

COBOL to Reality and Back

Because of Reality's convenient query and reporting language (called English on Reality, but other PICK systems called it Access), it was way faster to do reporting on the Reality system, so whenever management wanted a report from the mainframe, the quickest way was to get the data to the Reality system.

I wound up working with one of the First City IT people who knew COBOL to come up with a format that we could import into Reality. We had to do EBCDIC conversions and had to read the fixed width fields. This was a breeze compared with what I had had to do with the Terry Winter conversion, so I was always quick to create the imports for these. In some cases, I would create resulting data that would be EBCDIC converted and written to tape to go back to the IBM.

I wound up using the IBM terminals to track my time on the mainframe, and got used to using their messaging system.

Resetting Reports

First City had a printing system that was over 20 feet long.  They printed a ton of paper every day.  They also had to store a ton of reports. In an effort to reduce costs, they launched an initiative to figure out what reports were required, by whom, and were there other reports that would provide the information.

They managed to do some cleanup, but there were a large number of reports, some 100 or more pages long, that they were unsure if anyone needed them.

So they decided to take anything that they couldn't find an owner for and simply stop printing it, and see who complained.

Answer: No one...

I think they saved a few hundred acres of Amazon rain forest by that one initiative, reduced their storage and printing costs, and took an unnecessary load off of their computers!

Saturday, February 10, 2018

Pranks and Viruses

Over the years, at many places that I worked, the programmers I worked with were a fun bunch of people.  We'd have really intense discussions about technology and about how best to solve different problems, but ultimately, while we worked hard, we played hard, too.  Even the first viruses were more in the order of a prank compared with the ones we have today.

Changing User Prompt

One of the first pranks I encountered started out unintentionally when we were reverse engineering the Reality operating system.  We discovered a lot of interesting things, including that the very first byte (byte 0) of every frame (512 byte block of memory - that's how Reality organized memory) was not check-summed to ensure no corruption, but for one frame, it was actually used.  Both data and program space was organized by these 512 byte Frames. The first 12 bytes were used for linking and state information.  The rest was for data or machine code, for programs.

On PICK systems, the command line or shell mode was called Terminal Control Language (TCL).  The TCL prompt on Reality was a colon ":". For other PICK systems, they went with the greater than sign ">".

These first PICK systems, were what we called "Native" systems, where they provided the full operating system. Later versions ran as a shell on top of Windows, Unix, Linux and other platforms.  Prime Information was the first one to run as a shell, on PrimeOS, but it was the rare one. On these native versions, the first, lowest numbered frames were where the "operating system" and user assembly programs were typically loaded.  For most of them, frame 6 was the one that handled user terminal I/O.

On most PICK systems, byte 0 of frame 6 held the TCL prompt character.  If you had access to the system debugger, you could change that byte to any character that you wanted.

Be aware that while the TCL prompt character was ":", the BASIC debugger prompt was "$", and the system debugger prompt was "!".  These prompt characters would tell you that you were in different states.

When we discovered this, we changed it to another character to see what would happen. This was on a multi-user system, and we had other programmers doing work. As we changed it, we could hear swearing from the next room. We hadn't intended this as a prank, but it was too good to pass up! For fun, we changed it to the BASIC debugger prompt.  More swearing.  How about the system debugger prompt (this was getting too much fun!)  The swearing was sounding worried.  Then we changed it to the character that cleared the terminal screen.  This was too much!  The programmer in the next room came running over to say that the computer was having serious problems, only to find us laughing uncontrollably!  He realized he'd been had!

ERRMSG 3

On all PICK systems, there was a file that controlled the error messages.  While you could mess with any of them, by convention this was frowned on, with the exception of ERRMSG 3.  This ERRMSG simply printed the words "VERB?" and returned to the TCL prompt.  It was the error message you got if you mistyped a TCL command or intentionally typed random garbage at TCL and pressed return.

>ED ERRMSG 3
TOP
.P
001 E VERB?
EOI 001
.EX
'3' EXITED

>FOO
[3] VERB?

It was generally considered OK to make changes to this ERRMSG.  For most customers, this would be a simple change to something like "WHAT???" or "SAY AGAIN?", but for a couple of our customers, they were always on the lookout for something really elaborate or different to put in it. I saw some that were like 12 lines of text, suggesting you present yourself for a Bozo award.  Finally, one customer changed it to "BUTTERFINGERS!". Unfortunately, there was a worker in a warehouse of theirs who had really big fingers, and was convinced that it was personal. That was the end of them changing that error message!

Water Detected in Drive A

In the early days of DOS, I remember finding this program called "Drain". It was actually an executable called "drain.com".  You'd run it, and it would create what looked like a DOS prompt "A:\>".  The moment you typed anything, it would send a beep to the system speaker, and display an error message:

"Water detected in drive A"

The cursor would flash for about 3 seconds, then it would display another message:

"Starting rinse cycle"

Now spinning a 5 1/4 inch floppy drive made a sound a bit like a washing machine on rinse or spin cycle. Enough that you got the idea. It ran the rinse cycle for about 5 seconds, then printed out:

"Starting spin cycle"

This ran for another 5 seconds.

It would end with a message saying something to the effect of "Drive A is now dry - you may resume work." and it would drop to the DOS prompt.

My wife was volunteering as secretary at our church and she ran it, then called the pastor over, telling him something was wrong with the computer.  Great fun!

Update 2018-05-28: I found a link complete with screen shots!




Airplane Pilot Attempts to Outrun Electrons

One story a colleague of mine tells is of a software package that they had developed for airplane maintenance. The computer also had a word processor called JET and pilots would often use this to write letters.  They were at the Boundary Bay airport and a pilot was using the system as a word processor to write up a letter.  In typical style, still emulated by software programs today, if you typed the keystrokes that would delete your document, the program would ask if you really wanted to delete the document and you could type "Y"es or "N"o, followed by Return.  If you typed "Y" followed by Return, the program would tell you that document "xxxxx" has been deleted.

The pilot did the wrong keystrokes, and had typed "Y" + Return, and got the message saying his document was deleted, before he realized what it had asked him.  The quick-thinking pilot reached over his terminal (they were quite big affairs - see below) and yanked the serial cable out, then ran like mad to the computer room where he pulled the serial cables out of the back of the computer. Then he walked over to my colleague, Jan, who was watching all this in disbelief from another room, and asked her if she thought he was fast enough to stop his document from deleting!


The pilot reached over this and pulled out the serial cable on the back

Coffee Machine is Access Protected

One of the error messages that the computer would occasionally display was one that indicated that an operating system function had tried to access memory that they were not permitted to access or that was not properly initialized.  The message was to the effect that something "... IS ACCESS PROTECTED". It would then display the message "Abort @n.m". "n" was a frame number and the "m" part was the offset within that frame. This was the address of the program instruction that encountered the error.  It would then drop you into the system debugger, with the exclamation prompt "!".

Encouraged by the antics of other programmers that I knew, when working at First City Trust, I decided to create my own program for locking out my terminal while I went for coffee or lunch. I created a program that displayed what looked like a blank screen with a TCL prompt ":".  The first two times you typed anything into it, it would print the "VERB?" output, giving people the idea that they either mis-typed, or something else was going on.  On the third attempt, it would print the message:

COFFEE MACHINE IS ACCESS PROTECTED"
ABORT @12:00 noon
!

The exclamation mark "!" made it look like the system debugger.  At any of these prompts you could type a special password and you'd drop out to real TCL.  If you typed anything else into that last "!", it would display messages to this effect:

DELETE-ACCOUNT SYSPROG
ARE YOU SURE (Y/N)?

Regardless what you typed, it then displayed

DELETING SYSPROG
....

(SYSPROG was like the root account for these PICK systems.)

Finally the dots would stop displaying and it would log you off.

Note: No real SYSPROG accounts were ever hurt in the running of this program...


April Fools Day Endless Reboot

When I worked at Synex Systems, one April 1st, one of the programmers pulled a prank on all the other programmers.  The night before, he stayed late and changed every programmer's computer's autoexec.bat file to run a program that forced a coldboot just before finishing.

The effect of this was an endless reboot.  The first people arrived, started their computers and walked away. They'd come back with a coffee and the computer was still booting. OK, computers were slow at booting up (and many times still are), so they'd read a magazine for a few minutes. Finally, the realization dawned that this was taking too long, and that BIOS message they were looking at had already displayed twice...

You had to reboot off a floppy and edit that autoexec.bat file to fix it. That was a good April fools prank!

Stoned Virus

And then there was the stoned virus. It was one of the first real viruses, and very prevalent for a number of years.

When a computer boots up today, it looks for an operating system boot-loader in the "boot sector" of your hard drive. On the original IBM PCs and clones, the BIOS would start by checking drive A, then the fixed disk.  It was not uncommon for someone to load a program or data disk into drive A, forget it was there, and shut their machine down.  When they booted up the next time, they'd get an error message that the disk in drive A did not have an operating system on it.  You would remove the diskette and type Ctrl-Alt-Del to reboot.

The stoned virus was called a "boot sector virus". When you left an infected diskette in drive A, it would actually run a program that scanned your computer for diskettes and hard drives, and would add itself into the boot sector. Then it would display a message on your screen saying "This computer is now stoned." and it would print the standard error message saying that the diskette in drive A did not have an operating system.  You'd take the diskette out and reboot and all would look good. If you worked at Synex, you'd be convinced that one of those darned programmers put that "stoned" message on your computer as a prank.

Thereafter, every time you booted your computer, it would run this stub before running the operating system. Any diskettes you left in, it would attempt to write to the boot sector and infect it during boot-up. After boot-up, any diskette you put in, the now-infected operating system would attempt to infect with this virus.  Although I heard rumours that this virus would occasionally randomly delete files, to my knowledge, other than replicating itself, the only other thing I ever saw it do was occasionally it would display "This computer is stoned" message randomly during operating system boot-up.  These were the days of relatively benign viruses.

I knew of a company that published 6,000 diskettes to go out along with copies of Aldus Pagemaker, only to be told by Aldus that all of these diskettes were infected with the stoned virus!

We got in the habit of write-protecting all our floppy diskettes that we sent out to prevent our media from being infected.

This practice came in handy on multiple occasions. One such occasion was when I was at a large New York law firm, and someone came to me to say that there was something wrong with my diskette. I looked and he had an error message about not being able to write to the diskette. I pointed out that he should never write to our diskette, then got him the program that checked for the stoned virus.  He ran it and it turned out that every computer in the IT group had this virus!

Virtual Hand-Brake

And finally, in the same vein as the aforementioned airplane pilot, I leave you with an innovation that a number of us came up with, but no one ever figured out how to implement: The virtual hand-brake!

Have you ever had one of those times when you realized that the computer was causing havoc faster than you could undo it?  You need a virtual hand-brake!

I could probably go on for hours, but it's been long enough for today!  I hope you enjoyed this little trip down memory lane!