The Design and Implementation of laffer1

I just spent the last few hours updating firefox on MidnightBSD.  It's always a tiresome experience.  It's great to have a newer browser with some level of security available though.  mports/www/browser35 for those interested

One of the tasks I've been given at work is to write and maintain a newsletter generation tool.  Most people think of it as a simple application that sends and email with some canned text, maybe customized with the persons name similar to the "mergemail" feature in word.  

Well that's only part of what this application does.  It generates four different formats of newsletter, HTML, Text, PDF and a special HTML version to send with the PDF.  The PDFs are 50 pages long for one client everyday.  Today, I had to add stock price monitoring to the newsletter tool.  It checks the price of the stock on regular intervals (hourly) and includes the current price when the newsletter is sent out for this client.  I spent way too much time on this, but I have a feeling it will come up again.

Like everyone else on the Internet, I used Yahoo! Finance data as a source.   You can output a CSV directly from their web app and even query multiple symbols.  I limited it to hourly lookups and then only during market hours using a cron job.  This is probably the 7th cron job running.  They're all Java apps right now.  

I'm really torn on the whole Java thing with this. I think Perl or C would be better choices.  However, since the database schema changes constantly, I just used a jar file containing all the database stuff and pop it in when I do a schema change.  It really simplifies updating them.  

I hate spec changes, but it's common place at my company.  You can never get someone to tell you the whole story at any time or even half the story.  When you do get the story, you find it's changed from the week before.  

Anyway, the stock fetcher is pretty simple and slick.  I just used a Java Url class to fetch the data and then an open source library (LGPL) to parse the csv.  It's easy to parse CSV files yourself, but I didn't feel like taking the time.  

It runs through a loop and checks for each symbol stored in the database with a 30 second pause so as to not klll Yahoo's service.  

I had a fun bug today using html clean and jtidy together.  It seems jtidy was re-encoding the HTML entities again like ampersand so it was screwing up non breaking space, greater than, less than and apostrophe characters in the HTML output.  I have to run both because jtidy sucks.  It doesn't fix several types of invalid HTML.  If I ever get time, I'll try to fix it and upstream it. 

According to this BBC article, U2 brought in the most money last year. I can't speak for others, but the Chicago concert rocked.

I've noticed a new pattern among game developers.  Software has always been licensed, but now we're only buying unkown length licenses to these products.  With a game like World of Warcraft, I know I'm subscribing month to month, but I don't know if they will keep running the server indefinetely.  Eventually the game will die.  WIth a MMORPG, this is acceptible and inevitable.  However, other games like the new Assassin's Creed 2 require a constant Internet connection to play.  They actually download part of the game engine on each load.  If they decide to turn off the servers or go out of business, you lose access to the game forever.  This is the same problem that happened on the Xbox.  I bought an Xbox to play games online.  Soon, EA started turning off sports titles servers.  They want you to buy the new title each year.  That means I'm paying $50 a year to keep playing and the game experience changes each year.  Perhaps with an NBA game, I want to play the Pistons in 2004 instead of 2010.  I realize it costs some money to run servers, but this is rediculous.  Either they build in the cost to run the game servers for several years into the game or they warn people that the servers are only guranteed up for a year. 

I'm in favor of explicit minimums on the boxes (by date).  I want to know if the game comes out March 2010 that it will work until at least March 2011.  I also want to know in February 2011 that I have a month to play possibly.  It dramatically changes how much I'm willing to pay for a title.  If it's so important, they can add a sticker if they extend the time frame to the box.  This is only reasonable.  With other subscription models, I know how long I have.  SInce I'm obviously paying for server time now, I should know how long it lasts. 

I often buy games on Steam.  I know that if Valve decides to pull the plug, I could lose all those games.  It's a risk I've chosen to take.  WIth some games, I don't know how long i have now and in a retail box from a store like best buy, I expect to play the game for several years.  I still play age of empires 2.  I love it.  The game doesn't even run on 64bit windows, but I still play!  I even run 32bit windows just to play.  What if I like a game?  Will I be cut off forever? 

I started wriing this entry with a real world experience I had today.  Instead, I think I'm just take some constructive points from the situation.

1. When making a big picture roadmap with many IT projects, understand what each part is trying to accomplish.  You don't need the how for the first draft, but you do need the why.
2. Don't create three level deep diagrams of one component in the system that interoperates with other peices not even discussed yet. 
3. It's ok to talk about when you need a project complete, but don't start makin details timelines or final deadlines until the requirements are known.  You can't plan how long something will take when you don't even know what it is you're planning.  If you have a hard deadline, start with absolute necessities first.  Good software can always be enhanced.
4. Don't assume everyone is against your idea.
5. When you're asking someone to spend a great deal of money on a project, take the time to write down why you need it..  It really is the least you can do.
6. User stories are one way to obtain an initial list of requirements, but they do not replace a techical oriented view before one starts work.
7. When writing user stories, you must capture ALL of them or it is useless.  One user case left out could dramatically change the requirements for a project.  Programmers do write themselves into corners.
8. Reading one book will not make you a good project manager.  Reading this list won't either.
9. Don't trust any one website for information on software.  They can be wrong.  Some IT websites suggest products because vendors paid them or gave them an insentive. Others just have a bias.  For example, I might suggest  BSD because I prefer it to Linux.  That doesn't mean Linux isn't useful in many environments.
10. Programmers need to sleep sometimes.

I just read a blog for CIOs about planning IT projects.  In this blog, it suggested that CIOs or a "core" team plan deadlines for large projects.  As a software engineer who essentially is the project manager in my office, I found this article disturbing.  The reasoning behind the aritcle makes sense from a CIO's perspective given several other requirements.

The company must have one goal in mind with the project.  Personel cannot be "stolen" for other projects.  There cannot be fire drills every day.  Most importantly, the CIO needs to understand how long it will take to create a project.  A reasonable time frame must be defined.  My experiences have shown that management has no idea how long projects take.  That poor, young project manager might not have a good idea how to do estimates yet, but (s)he does know how large a project is. 

In some companies, this approach would work well.  It will not work in all companies. The approach is getting used more frequently in my company.  The net result is dropping everything and putting everyone on a task.  It does get that one task done as quickly as possible, but the quality of the project suffers.  Further, it puts every other project behind schedule.  Sometimes project managers are right.  Things happen during development.  Odd bugs pop up. New requirements are brought in during development. 

While I'm on the subject, it's also important to have clear goals for a project at the beginning.  This approach reminds me of the waterfall method.  WIthout a very clear, well thought out specification, large software projects will always fail or at least be delivered well after the due date regardless of who sets the timeline.  You get some leway with agile methods, but you still need to know what you're trying to make.

I agree with one point.  It's important to have clear goals defined for your IT staff.  Tell them what you need this year.  Give them time to implement long term solutions.  It will save you time and money.

I better stop here. 

HTML Cleaner 

JTidy

TagSoup

JTidy cannot create valid XHTML strict pages.  A combination of HTML cleaner and JTidy cannot make valid XHTML strict pages for some input.  double br tags, some attributes like height, and duplicate id attributes cause problems.

http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-0232 

http://www.microsoft.com/technet/security/advisory/979682.mspx

Microsoft has issued a fix.  This was for the previously mentioned NT 3.1 through Windows 7 vulnerability I talked about in my blog. 

About ten years ago, dnssec was invented to deal with a problem plaguing the Internet.  There is no trust in the DNS system.  

Background

Many people might have heard the term DNS, but never thought about what it is.  DNS, or the Domain Name System, is the process by which a domain name like midnightbsd.org is translated into an IP address 70.91.226.201.  Without this system, one would need to type in IP addresses to access websites, send email, or chat online. 

The system was invented at a simpler time when people trusted each other on the Internet.  This was before worms, massive spam, or websites.  

Today, many people try to impersonate others on the Internet or worse yet, their websites.  You could create a DNS poisoning attack so that a user accessing a DNS server to lookup google.com is redirected to a fake site.  This site could log information and pass requests to the real google.com through a proxy.  The user may never know the difference.  

Systems like DNSSEC validate DNS queries by a trust relationship.

Using DNSSEC

Individuals don't need to do much to use DNSSEC aside from purchasing updated software.  Windows 7 had DNSSEC on it's list of new features (not confirmed it was added in final builds).  The client (your computer) must be able to understand DNSSEC queries for it to be of any use.  Otherwise, it is simply ignored.  

System administrators must enable DNSSEC on their DNS servers (resolvers) as well as on zones to get the full benefit.  You can think of a zone as a domain name.  Things can be further divided into sub zones such as .com vs midnightbsd.org.  

Enabling DNSSEC on BIND 9.4+ resolvers

In options: 

        dnssec-enable yes;

        dnssec-validation yes;

        dnssec-lookaside "." trust-anchor "DLV.ISC.ORG";

Microsoft is patching a 17 year old bug in Windows that affects WOW (not the game) in current versions of Windows that allows it to run old programs.  The BBC is reporting it's a bug carried over from NT 3.1. 

I wonder what other goodies are hiding in our copies of Windows.

I just read that Cayenne 3.0 RC2 is out.  Cayenne is an ORM.   I've been using it for almost a year now.  It's a very easy way to do data access in Java.  There are a few quirks.  Most people love Hibernate and don't consider alternatives.  Cayenne is very easy to work with.  It has a client gui to configure and setup your mappings or you can write an XML file by hand.  

This is caused by using two apostrophe's escaped in a row in an RSS feed.  I can't find anything that says that's invalid.  I even tried switching from the ' to ' 

I bought a new HP server for hosting my websites and email.  I've spent the last day trying to get the system into shape to run MIdnightBSD current.  So far, the NIC isn't supported, the DVD-ROM drive causes some issues with shutdown in "Compatibility mode", I have to disable turbo mode to avoid an inturrupt storm (Xeon 3430), and some of the CPU features aren't detected properly.  Since, 0.1.1 couldn't boot and 0.2.1 amd64 wasn't working well, I had to partition from 0.2.1 and then run a current live cd, copy some files with cpdup, and then chroot it and run make buildworld to get it ready to go.  

At this point, I'm running with USE_MPORT_TOOLS and starting to install ports.  Chris's mport tools have been working pretty well on my desktop; I figured it was time to run them on the server.

The system currently only has 2GB of RAM, but I figure I'll be upgrading that later this year.  I bought a SSD last month for the OS boot drive and the 160GB drive is for /home and swap.  I considered var on there but since i only run email on it, i figured it would be tight but OK.  

The CPU is awesome.  I've been pro AMD lately, but I must say this Intel chip is quite nice.  I'm glad Caryn talked me into it.  

I've found a number of bugs with MidnightBSD while installing the OS on the server so far.  I've got a lot of work ahead of me. 

I purchased a copy of VMWare Fusion 3 for my iMac today.  In the past, I've used Parallels for intel Macs.  While there are a few features I miss, I must say that VMWare is quite fast.  I haven't found a way to run concurrent VMs like Parallels can do, but I don't use that feature often anyway.

I have not tested it with Windows yet.  Most people use it for accessing their bootcamp partition or running Windows applications.  I use it to test MidnightBSD software and to work on ports.  I have a real PC for windows.  

The updater is nice and it has tools for Windows, Linux, Solaris and FreeBSD.  I can't get the FreeBSD tools to install under MidnightBSD.  There are many levels of OS version checks and I'm missing one somewhere.  X.org worked perfectly under MidnightBSD 0.1.1 i386.  I'm updating to current now.  

The networking code works really well.  It seems much more stable so far.  I'm running on Snow Leopard.

I've had mixed feelings about VMWare products in the past.  The windows versions have caused BSODs on me.  Recent versions of VMWare player have been a lot more stable and based on that I decided to give it a shot for my Mac. I've also used quite a few of their free products over the years.  Lastly, I found that Parallels hasn't been innovating much and also not putting enough testing time into other OSes.

o 30+ new Nmap Scripting Engine scripts
o enhanced performance and reduced memory consumption
o protocol-specific payloads for more effectie UDP scanning
o a completely rewritten traceroute engine
o massive OS and version detection DB updates (10,000+ signatures)

This looks awesome. In case you're not familiar with nmap, it allows you to scan a host to determine the OS it is running and look for services by way of a port scan.

http://wiki.xtronics.com/index.php/Raid

See the difference? 

After my little rant, I decided to backup my claim about FreeBSD.  Here are some articles on setting up gmirror in freebsd

http://www.freebsd.org/doc/handbook/geom-mirror.html

http://www.freebsddiary.org/gmirror.php

I just went through a day of hell today; it's name was Linux.  Ubuntu server does not properly support RAID either with "fake" raid as in the Intel Matrix RAID controller on my core i5 development server at work or with the Linux kernel's built in RAID.  Ubuntu cannot detect if a drive has failed, read from either disk for performance improvement or even detect their was a read failure and silently switch.  Your box has to crash for you to know a disk failed.  Worse yet, it doesn't even install.  It tries to use EXT4 file system which does not support GRUB.  GRUB2 can't work with RAID yet which is the default.  So Ubuntu server is a useless piece of shit.  

So after wasting time with three installs of Ubuntu server, I went on to look for another Linux distro with working RAID.  All of them required insane hoops to get it to work and each distro was different.  Suse has a nice gui installer, but with a DVD download and several duplications of partitions, it was just not what I wanted to deal with.  FInally, we ended up with Debian which is working ubuntu.  (it's the distro ubuntu is based on)  The RAID support works in Debian but requires some rediculous steps.  Vamsi installed debian (first) and then we had to make an identical partition table on the second disk, initialize md for EACH partition on that disk, rsync over the files (we had 3 partiitons including swap) , fix GRUB (not GRUB2), and /etc/fstab, reboot, then add disk 1 to the array (each partition actually) and then wait for Linux to copy all the shit back over from the second disk.  Not to mention fixing the md config file.

In FreeBSD, this would have been a 10 minute job including installing the OS.  Even Windows or Mac OS X server can do this without a problem.

Conclusion: LInux is not ready for the desktop or server room 

http://schemasync.org/

This tool lets you "diff" mysql schemas and generate a script to update an old database to a new one!

http://stupidfilter.org/main/index.php?n=Main.About

Someone sent this to me at work.  It's a project to write a filter for "stupid" english.