Digiwar - the Yeep-blog

Like I wrote last time, my PHP-program ran like a charm and was great. Until marketing decided they wanted to change the prize plans on the product. This change required the billing of the product to be done on another billing system and because that billing system was so tightly integrated into everything, it would also require the LDAP to change and we’d need a lot of additional information we didn’t need previously stored in a new database. But, on the bright side, the processing should be simpeler, because now most of the business rules would be implemented in that billing engine and not in my PHP-program anymore and the source database could now (after an update) send changes to existing customers (or new customers) instead of giving us a complete database dump like they did previously.
It looked like life had become simpeler.

Then it was decided that to make the migration easier, we would create a version of my PHP-program that would be somewhere between the old version on the new version. It would accept a complete database dump like it always did, but that dump would be augmented with the additional information we’d need. And that version would already talk to the new billing engine like the new version should (using SOAP). So basically we’d have a version with all the ‘difficult’ old stuff and all the volumious and ‘difficult’ new stuff. The reasoning was good. It should, in theory, make the transition easier, but then again, theory and practice are two different things altogether.
What happened was that with the extra information and the increased userbase (now more then 50,000 users) the memory usage of the program started to exceed the ammount of RAM installed in the machine. Not a problem per se, but the program wasn’t the only program running, so near the end the OS would kill the process because of memory problems. And also, due to hard disk swapping and 1 or more SOAP calls per records (in addition to an LDAP query per record) to program ran slower. A lot slower. It went from 45 minutes to over 14 hours at it’s slowest!
This was a problem.

Fortunatly these problems only got realy bad when I had almost finished the final version of the program. That version would only use changes so the memory usage would be very small and because the amount of changes is relatively small the slow-down because of the SOAP calls is less severe. It now typically runs somewhere between 30 minutes to an hour on an average of 1,000 to 1,500 records. And that’s when it runs once a day, if we double this, the amount of records to process per run are halved with the speed of the run increasing a little more then 100%, so we have some breathing space.

Ofcourse the final version wasn’t perfect. In fact, it was a nightmare. Part of it was due to incorrect specs, part due to missing knowledge on my side (some ’special things’ about PHP I didn’t know) and part was due to the test-process being inadequate.
So what did I learn from this? Well, if I’d had a better development enviroment to program in, lot’s of bugs could’ve been avoided. It can’t debug PHP like I can debug C++ code on Windows. I know there are PHP-debuggers, but I couldn’t get them to work and besides, even if I could, they would be no good, as my desktop wouldn’t be able to reach the test-platform. I could have worked around that though.
No, the biggest problem was that I should’ve figured that the complexity of the program had increased and PHP wasn’t the best solution anymore. I think that in the future I should not be affraid to rewrite part of a program in another programming language, because the added complexity of the new features will make the program harder to make and maintain in the current used language.
If they ever need more complex features, I’ll rewrite the program in C# for sure. That way I can program and test in on my desktop and run it on our application servers thanks to Mono.

There is this pogram at work, the one I described in an earlier post, that I created using PHP. Now, I’m a C/C++ programmer, so I could use PHP very easy, because it looks so much like C. It even behaves in a similair manner. There are differencess though. Like how you don’t need to allocate and free memory in PHP, or how a variable can contain anything. In PHP a variable has no type. I can assing a string to a variable and then later assing an integer or an array to the same variable. That’s also very handy. But, as it turns out, there are also other differences. Differences I didn’t know about and that caused bugs in my code.

Here are 3 differences that caused me a lot of headaches:

1. A variable containing an empty array equals to boolean false.
2. Using continue from a switch statement, works like a break and does NOT continue to for-loop the switch was in.
3. Using unset() on a ‘pass-by-reference’-variable only destroys the local copy of that variable, even though ‘pass-by-reference’ means you don’t create a local copy.

All of these things are documented, I just assumed they worked like they would in C/C++. So it’s all my own fault.
Still sucks though.

A little over a year ago I was assigned to a new project. The company I work for was going to start a new ADSL service and we needed to connect three totally different servers together. On one end was a database where all the customer information was stored and on the other end is the LDAP our RADIUS uses to authenticate users. In between is a database that controls our IP-address assignments. To make the problem even bigger, the customer database was maintained by a complete different division and the only interface to the IP-address database available is a PHP-class to be included in a PHP-program.
We solved the first problem by aggreeing on a way to get the necessary data to a shared place somewhere in a CSV format. All that was left for me was to decide which programming language to use for the program that would do the required processing.

Some people base the choice of the programming language on the requirements of the program to be written. I don’t know that many programming languages, so my choice is already limited. Besides, I am dependent on System Administrators en Application Administrators who must provide the enviroment for my program to run in and it’s almost certainly not going to be the only programming running on the server, so it shouldn’t need some vague library version that causes problems with other programs.
So what are my choices?

1. Shell script
2. Perl
3. Java
4. C/C++
5. PHP

Let’s reason and see which are actually viable:

My shell script knowledge is really basic, so there is no way I could use it to do the processing I need. I also see scalability issues. So no Shell script.

I probably know more about Quantum Physics than I know about Perl, so that’s a big problem. Due to time-constraints I can’t ‘learn on the spot’. Pity, I think Perl would’ve been a real good choice.

I also don’t know Java. A lot of my colleagues really like it and I do know C++, so I might be able to learn it and use it. I do not have any experience with it, or with the development enviroments available to me, but it’s still an option. There is still the problem of using a PHP-class, but I can work around that.

C/C++ is the language I program most in. I know it a lot, although I use it almost exclusively on the Windows platform, so I’m not really familair with the Unix intricicies. Also, the development enviroment we have for C/C++ development on Unix sucks extremely. Especially for someone who is used to the clever-goodness that is Visual Studio 2003. But for C/C++ we have library issues. Sure, we have an LDAP library we can use, and I can probably find some FTP library and some RegExp libraries, but it’s all additional work and I’ll need to learn to use the libraries as well. Then I’d need the libraries to run on Solaris (our development server) and on Linux (the server the program is eventually going to run on). It’s possible, but it’s going to be a big time-consumer. And we don’t have that much time (as usual).

Then there is PHP. I know basic PHP and it’s very similair to C, so it’s not hard at all. It has great built-in support for a gazzilion different features, including FTP, LDAP and RegExps. And it’ll be a no-brainer to use that PHP-class for the IP-address database now. The business-rules are not that hard and there are a lot of free PHP development enviroments I can try out.

So PHP it was. And it was good. The program ran like a charm. No problems, it processed over 40,000 records in under 45 minutes and it was great.
Then….things changed…but that’s for part 2.

True, so true.

A few weeks back I got an e-mail from my hosting provider, telling me they updated their nameservers, so I should change the one on my domain. However, I maintain my DNS settings at my registrar, sio I ignored the e-mail. Until I noticed a 404 at my RSS feed. I visited my website and all I got was the “You’ve done it!” Apache standard page. Woops!!

Some nslookuping and editing of the host file and I found my page again, but the 4 blog-entries I made the last few days where gone! Good for me I’ve subscribed to my own RSS feed, so I can easily restore them.

That’ll teach me to ignore those e-mails in the future!

So, one of the first things I learned using C# to do some LDAP queries is that when you create the System.DirectoryServices.DirectoryEntry object and you specifiy the protocol to use (in my case LDAP), the text-string in which you specifiy it is CASE-SENSITIVE! Imagine my amazement when nothing in Windows was case-sensitive and all of a sudden this is!
“ldap://myldapserver” doesn’t work, while “LDAP://myldapserver” does. Retarted!

Another thing. When you connect to an Active Directory, it imposes a result limit of 1000 records. Meaning you can’t get more then 1000 records out of a single query. It took me a few hours to figure this one out, but it turns out it’s really simple!
The System.DirectoryServices.DirectorySearcher object you use to do the actual searching, has a property called “PageSize”. Setting this to 1000 (or 999 as I do, just in case) solved that problem, because the object will then get the results of the query in batches of 1000 (or 999) and add them together for you to use later on. So to you the programmer it’s like you receive all the records in one call and to the LDAP server it looks like the call has been segmented into smaller calls each receiving 1000 (or 999) results, staying nice below the server imposed limit.
You should know, however, that when I set the PageSize to anything but the default, I couldn’t query our Netscape LDAP. For some reason that LDAP doesn’t understand the PageSize property.

A few weeks back I bought a C# book by O’Reilly and started reading it. Turns out C# is a pretty cool programming-language. And since all kinds of new projects at my work involve (or will involve) Microsoft technologies, it’ll be particulary useful for me to know some .Net programming.
I wrote some small and simple C# programs to help me out with some small menial tasks at work, but to really learn the language I decided to try and write a real application with it. Recently my interest in XMPP (Jabber) flared up again, so I decided to try and write an XMPP-client in C#. It’s not supposed to be for general use, it’s just a learning project for me. I can learn both C# and XMPP doing this, both technologies which can really make a difference in my line of work.
So expect some of the things I’ve learned to show up on this blog and maybe, really really maybe, there might one day be a downloadable XMPP-client on this site. Don’t hold your breath though….

[Last played: My Life With The … - Starmartyr]

As you’ve read by now, or will read over the course of the next few weeks, Microsoft announced it toned-down it’s feature list for Longhorn in order to make a 2006 shipping date. Most of this stuff was already publicly know even though it had never been confirmed by Microsoft. And now it’s official. However, you should read Robert McLaws blog to see what’s really going on in stead of reading the hype on biased news sites.

It’s been a long while since I posted anything. It’s not that I’m no longer using the site, it’s just that I’ve had a lot to do this summer (and a lot I wouldn’t do, because I wanted to enjoy my free time), so posting on this site was a last-priority. I will start doing so as of right now, so keep in touch.