The Design and Implementation of laffer1

I found additional links that I used to work with the new hard drive after Thanksgiving.   I should explain what the problem is with these new drives.  For a very long time, the standard sector size was 512.  It's been that way so long that computer bioses, operating systems, and software programs assume this to always be the case.  Due to legacy issues with DOS and tricks used to increase drive capacity over the years, many hacks have been implemented that further limit options.  In order to pass the 2TB barrier, drive vendors have decided to make the drives use 4K sectors internally and then expose 512 sectors to the OS.  The drive has a firmware that converts between the two.  WD calls these advanced format drives.  Since the OS is lied to, it's hard for it to properly align partitions on the drive.  What does alignment matter?  If a partition is created on a boundary, it lines up with how the drive reads and writes data (in blocks).  This makes it run fast.  If it has to write to two blocks because of an alignment problem, it slows the drive down significantly.  Windows XP can't handle these new drives without some hacks.  Windows Vista and 7 can if the drive is formatted by them.  (not an upgrade from XP)  

Linux until 2.6.33 can't fully handle these drives.  FreeBSD can't handle these drives without manually partitioning to set the alignment.  It's a pain.  

More on the problem:

4K Sectors & Glabel patch info

WD EARS howto on FreeBSD forums

WD KB article on their drives

Microsoft performance & drive partitioning for SQL server

4K in Linux

 Anything before 2.6.33 doesn't support 4K very well.  Distros with tools dated after April, 2010 can handle alignment of 4K drives properly.

UFS & 4K

Performance issues exist with the defaults selected in FreeBSD.  Matt Dillon made some comments about what to do with UFS to get it to perform well. 

DES test program for FreeBSD

This program lets you test performance differences with different alignments (sectors) of the drives in FreeBSD and MidnightBSD.

GLabel patch for FreeBSD 9-CURRENT

This patch makes Glabel align properly when using a certain flag for 4k drives.

I've been a bit confused by anonymous functions versus closures.  Many references use them interchangeably.  They're different concepts.

An anonymous function is like a regular "named" function typically.  Some languages put limitations on them, but most of the time they're just like regular functions.

a = function() { 

return 1;

}

This isn't useful, but it's an anonymous function assigned to the variable a.  It can be helpful for event driven programming in javascript and other languages or any type of callback.

In contrast, a closure is just a function that has access to the parent environment.  Basically, you can access variables in the parent namespace. These variables are called free variables.

function hello() {

var x = 1;    

    function world() {

         return x + 2;

    }

return world();

} 

In this case, a call to hello returns 3.  

You can also combine the two which is what most people do:

function outer(y) {

    a = function (z) {

           return z+y;

    }

   return a(3);

}

outer(4); // returns 7

Finally, I should mention anonymous functions are sometimes called lambda.

 I'm contemplating following up on a job lead in Seattle for a large online retailer.  It would be a very cool job working on java and C++ backend code.  Of course there's the house and other factors as well as caryn's job.  Ignoring work factors, Seattle has a large BSD community and coffee!  It would be a big step career wise for me.  

My first week at work went ok and this lead is very similar to what I'm doing now although it's PHP and for a smaller online store.  

I managed to fix up to 60 ports in the last few days for midnightbsd.  I'm validating that with a magus run now (the midnightbsd build cluster software). 

My plan this weekend was to work on my little project, but I've had all kinds of weird distractions.  I'm a bit eager to try out some HTML5 soon.  I just got through the spec early this week and I haven't worked with many of the new tags yet.  (article, section, etc)  I decided converting an existing site doesn't make much sense yet.  

I just renewed foolishgames.net for another year.  I'm not using it much anymore but all my DNS servers are setup on that domain.  

I've been trying to track down a BSOD issue on my vista machine.  I updated the video and motherboard chipset drivers yesterday in hopes or resolving it.  It's been crashing in ACPI and nvidia's video driver.  The system was stable when I was running ATI cards in crossfire, just very slow.  At the same time I put in the video card, I also replaced the power supply.  It's hard to know which one is doing it.  It could even be extra heat from the new video card.  It only seems to do it when it's been on a long time.  (by long i mean a day or so)  I've also seen it crash in BSD, but I that is the cheap IDE raid controller which I don't use in windows.  

Later today, I'm driving to Southfield for some final paperwork before I start my new job.  I'm happy to be starting a new gig.  It took me a week to find it which is amazing with all the things I hear on TV about people looking.  I'm very lucky.  

Apparently Caprica was cancelled.  It was a prequal of BSG.  I'm not shocked because it took 3 years to show the first season.  WTG NBC/SyFy.  

Last Friday was my last day at PRIME.  I met with several former coworkers at Grizzly Peak for some beer.  Caryn came about half way through and we had a lovely dinner out afterwards at Cottage Inn.  It was a very relaxing night.  

Many of you know why I left PRIME.  I did enjoy the programming and some of the administration work.  I miss designing large systems and having employees work under me.  I don't miss the upper management style.  I think I'll leave it at that.

I had a coworker contact me today about a configuration setting on the dev server at PRIME.  It took almost a week before they wrote me an email.  I'm impressed.  

I am a bit curious how they're doing, but it might be best I don't know.  

I'm currently looking for a new developer position.  I'd prefer something with Java, C#, C, C++, or Visual Basic.  I don't really know Objective-C well enough to go that route.   I had a phone interview and scheduled another for Monday. So far, I've got my resume out to 17 companies for various development positions.  

Some of you might not know what I did at PRIME.  I worked on web applications (websites) that collected article data and evaluations on content from various sources.  This included news outlets, magazines, websites, blogs, twitter, and other RSS feeds.  The content was then stored and analyzed or placed into a newsletter tool that I also wrote.  Newsletters were sent out daily to fortune 500 companies in HTML, Text or PDF format.  I find it rather crazy that CEO's of large companies you've heard of were looking at newsletters my software made.  I also managed 9 servers including Linux and FreeBSD systems and managed the IT resources in Ann Arbor.  It was like 3 jobs in 1.  I went from intern to senior software engineer in 1 year.  It was a wild ride.  

I had some interesting requests like storing every tweet on twitter or 1 million articles a year including translations in 6 languages.  I developed software that could scale close to that, but the complexity of the system made it hard to scale out and limited resources prevented scale up as well. I had many ideas for optimizing the system and even more to improve the work flow.

I'm not under an NDA, but I don't think it's right to get too specific in a blog.  Let's just say I wanted to migrate from hundreds of distinct data sources to a centrally managed system and there was a lot of resistance. 

I just noticed a security vulnerability in T eamSpeak, a voice chat application used by gamers.  Apparently one can gain system level access through the TS client. 

While you're having fun, someone can gain admin rights on your machine.  How festive.

 Just got out of an IT meeting at work.  It's been a rather slow day.  It was a confirmation of the transition... blah blah.  

I got time to test chrome on FreeBSD.  It works great.  Much faster than Firefox, but no flash support.  

Got a free lunch on the company today at Connor O'neils or whatever..  gotta love that.

I just read a PC World article about the death of Linux on the desktop.   The author believes desktop Linux is dead because of lack of content.  My first thought was software.  There are many areas Linux is behind with applications.  However, upon considering his words I realized he meant music and movies.  He made references to DRM, etc.  

Thinking about my own setup, this is a huge problem.  I have over 400GB of iTunes content.  Much of it is video.  I would not be able to play it on Linux.  I can't stream netflix movies on Linux.  I can't play games I purchased on steam on Linux (for the most part).  

Most people don't have large iTunes libraries.  Many people don't have netflix accounts.  These would stop me from switching exclusively to an open source OS, but it does not stop many others I know.  My mother only buys music on iTunes.  Most of that can be upgraded to DRM free versions and then moved to another platform.  Real player on Linux can play AAC files.

So the question isn't what stops a tech savvy geek who has a lot of toys on his PC, it's what stops the average person.  The simple answer is availability.  When Asus eee PCs shipped with Linux, people bought them.  Some people didn't even notice it wasn't Windows.  Others returned them because they couldn't run a Windows application.  The failure was not clearly advertising it wasn't windows compatible and using a non standard Linux distro.  Had Asus shipped Ubuntu and their massive package repository, one could have put on the box "4000 applications for free".  

The Apple iPad is proof it can be done.  It doesn't support Flash and people still use it.  It doesn't play netflix movies.  It doesn't have large storage capacity for iTunes (although it can play the files).  Another example of success with Linux is the Google phones.  People don't even know they have Linux in their pocket.  

I've started to write the web application phase of the rss mixing project I started awhile back.  Mostly, I focused on building security and basic classes for web pages today.

I see two pieces in the first version:

1. Search of existing content

2. Feed mixing with per user hash table of feeds. 

Bad libc glob(3). 

 http://www.h-online.com/open/news/item/Flaw-in-libc-implementation-threatens-FTP-servers-1103319.html

http://www.google.com/safebrowsing/diagnostic?site=

Fill in the url you want to check at the end of this URL.  Google will report if a site contains malware or not. 

So I've been playing Starcraft 2 a lot lately.  I'm at a point I can play vs ai and win on medium with 1v1, 2v2 or 3v3.  I'm still working on 4v4 and "hard" on 1v1.  After losing a few times, I spawned as the Terrans and slaughtered the zerg on medium.  It's sad I can invade with 4 cloaked banshees and take out half their base. 

 I started reading Autotools by John Calcote.  It's a book about the GNU autoconf, automake and libtool utilities. I've found a few questionable statements in the book, and the author tries to justify his view.  I don't view it as a bad writing style, but I shouldn't need the author to convince me of his geek-ness.  I get suspicious of sources that feel they need ton convince me of their credibility.

Obvious errors:

The author refers to GNU/Linux as the proper name for Linux.  This is disputed in the Linux community.  I've been corrected while using both uses.  From my current perspective, Linux is a kernel and GNU/Linux is the kernel + GNU tools.  Since the official GNU kernel is GNU Mach/HURD, it doesn't make sense to claim Linux is a GNU product.  It's not.  

Claiming one is an open source fan while writing a book on OpenOffice + WIndows 7 is silly.  I have no problem with the author using WIndows for gaming, but it doesn't help credibility.  On a certain level, I find the choice of Open Office to show poor judgement.  If you're on a platform with a real word processor, why use Open Office?  It's slow, error prone and nothing but headaches. Anyone who's used OO on any platform knows what I'm talking about.  One could argue LaTex or docbook are better choices for proving one's worthiness as a geek.  

Trying to convince the audience why the book is needed is silly.  I bought it because I wanted a book on autoconf.  

Finally, Mac OS X applications do have a use for such a script.. it's called GNUstep.  

One of the problems on the web is all the different character encodings.  Computers represent information in different ways.  Some of these approaches handle multiple languages, others do not.  One such encoding is UTF-8.  You can test for UTF-8 in your web applications using this regular expression:

http://www.w3.org/International/questions/qa-forms-utf-8

$field =~
  m/A(
     [x09x0Ax0Dx20-x7E]            # ASCII
   | [xC2-xDF][x80-xBF]             # non-overlong 2-byte
   |  xE0[xA0-xBF][x80-xBF]        # excluding overlongs
   | [xE1-xECxEExEF][x80-xBF]{2}  # straight 3-byte
   |  xED[x80-x9F][x80-xBF]        # excluding surrogates
   |  xF0[x90-xBF][x80-xBF]{2}     # planes 1-3
   | [xF1-xF3][x80-xBF]{3}          # planes 4-15
   |  xF4[x80-x8F][x80-xBF]{2}     # plane 16
  )*z/x;

http://www.phpwact.org/php/i18n/charsets

The latest in zend bugs.. strlen reports 27 characters for a 10 length string.

http://ha.ckers.org/xss.html

I've been working on a new web application to download and mix rss feeds.  The idea is to cache feed items into a database and allow mixing of content from many sources.  I have about 1300 rss feeds so far on a variety of topics.  It's rather interesting to do like queries on the tables and find all sort of information.  

My long term plan is to have it index all public just journal entries, and content from other sites like twitter.   (just popular stuff)

There are services like this already, but many of them charge a lot of money for the information.  It's not like a typical search engine because one wants to access the information repeatedly and google tends to block that.  It's an interesting problem we stumbled onto at work.  Most likely content will go pay down the road, but I think there will still be free content as well.  

The first version of the rss fetcher is complete.  It's populated about 444,000 articles so far (rss items), 56000 categories (per rss 2 spec), and 55000 enclosures (file attachments like podcasts and images).  

One service actually charges 75,000 dollars for this functionality. They have a lot more content than I do, but most of it is garbage from twitter.  If you do a keyword search for say clorox, it ends up with posts about bleaching blood out, throwing it on people, cleaning tips and other crazy things.  There's obviously a need for good filters and smart content searching.  I only know of one method to do this right now and I'm not going to buy IBM Omnifind :) 

This is one of the many problem domains I deal with at work.  Thing is, they have no interest in getting the content themselves whereas I see a lot of potential in it.  This doesn't really overlap with work but it's related to what I do right now.  Scary isn't it? 

The biggest hurdles are:

1. Bandwidth.  Downloading RSS feeds takes a long time.

2. Storage capacity.  I'm not sure how long I can retain content. 

3. Blocks.  I might get black listed harvesting so care has to be taken in fetching content.  The java libraries I'm using right now don't honor rss feed intervals, but I'm limiting to at least 60 minutes for now.

4. Legal.  I'm not planning on charging for this data now and effectively i'm acting like a search engine.  I spider content, collect it and cache it for a period of time.  The only difference is how one accesses it.

The next step in the project is creating the website.  I've got help with that phase.

A friend of mine has been working on merging useful patches from Debian and MirBSD for CVS into a new port for mirports.  This is interesting and we might look at it soon.  It adds a number of useful additions to CVS.

MidnightBSD CVS is based on a patchset from DragonFly BSD. It's newer than FreeBSD's archaic CVS version. Newer versions have some issues with CVSWeb, but ViewVC works fine.