MidnightBSD Developer Journal

 MidnightBSD gains the wake(8) utility.  You can use it to wake up machines supporting WoL on the network. 

 If you're running BIND on MidnightBSD 0.3 or lower, I strongly recommend switching to the mports version. There are a few security issues with the base system bind.

I've patched it tonight in 0.4-CURRENT, but it's not fun to backport.  

Also, our resolver has been patched to allow underscores in names.

 I just patched  CVE-2011-3336

This could be used as a denial of service attack against FTP servers or anything else that uses regular expressions from libc.  A "bad" regex can consume massive amounts of memory.

The fix has been committed to CURRENT.  I want to test this further before applying it to 0.3, but if you have a public facing system, you may wish to grab the following file and rebuild libc:

http://www.midnightbsd.org/cgi-bin/cvsweb.cgi/src/lib/libc/regex/regcomp.c?rev=1.3;content-type=text%2Fplain

The patch does pass the regression tests, but I'm not certain the memory limit calculation is correct. Our implementation is a bit different than NetBSD's as it supports wide characters.

Over the last few weeks, I've been working on several new projects.  I ported the FreeBSD 7.1 ULE scheduler to MidnightBSD along with CPU affinity and the cpuset(1) utility.  The default scheduler has been changed to ULE on i386 and AMD64.  

MidnightBSD now supports booting off of GPT partition schemes and gpt(8) was updated to allow one to partition the disk.  sysinstall does not support this, however.  

Marcus von Appen's BSD licensed cflow implentation has been imported into MidnightBSD.  cflow allows one to build call graphs from C and Assembler files and is part of the POSIX.1 standard.

less v436 and awk 20110810 have been imported into 0.4-CURRENT.  They are the latest versions available. I've also updated the tzdata to v2011l from October 10, 2011.  The timezone data is now hosted at ICANN.  

et(4) has been imported.  It is an Agere Gigabit Ethernet and Fast Ethernet driver written for DragonFly.  I have not included this in the GENERIC kernel, but one can use the kernel module.  As I don't have hardware to test with, feedback is very helpful.

A security issue with unix domain sockets was fixed and the linuxolator updated to handle the changes.  

fetch(1) was updated from FreeBSD 9 sources.  It includes several changes, but the most important one is that passive mode FTP is now the default.  If you wish to use active mode FTP, you must set an environment variable now.  

gcache(8) was introduced to speed up RAID 3 volumes.  

alc(4) includes several fixes which improve performance.  I've also added some workarounds for users with CyberPower UPS devices connected to their systems over USB.  

My next big project is attempting to get a live cd based installer working.  This will also need to work with the new mport tools.  Boot crunch barfs on liblzma (xz) which makes building a crunched sysinstall impossible.  libmport uses lzma archives to store mport packages now.  This is the biggest issue holding up the next release. 

MidnightBSD 0.3-RELEASE-p4 includes a patch for a regression introduced in p3 with the linuxolator.  The unix socket fix changed the size of a kernel structure and it no longer matched the linux computer part.  This broke compatibility with Linux applications.  

Users are advised an ugrade to this release. 

mports

 PHP 5.3.8 and phpMyAdmin 3.4.5 are now available in mports.  A new magus run will be scheduled for this weekend to test the recent mports changes. 

Refinement continues on the mport command line tool.  Current work focuses on integrating libmport with the installer so that we can do the next release.

 I've been working on getting ports in shape for the next release.  Python 2.6.7 is now in ports along with HelixPlayer (Linux) 11, Gstreamer 0.10.35, and emacs 23.3a.

I've found myself sidetracked lately with server upgrades, a new laptop I'm trying to get working with MidnightBSD, and other fun things.

Several new happenings in MidnightBSD:

xz 5.0.3 in CURRENT

improvements to mport, msearch and batt(1).

In mports, apache 2.2.21 and xz 5.0.3.    

The last magus run is looking a lot better.  Here's the results from i386 0.4-CURRENT:

The mport command now supports a new "download" command.  This allows a user to download a package independant of installation.  It's saved to the normal package storage path inside of /var. 

This is useful when you know you're going to lose network connectivity and want to fetch several packages for installation later.  

This past weekend, I've focused on updating ports.  www/linux-firefox-devel (3.6.21), most multimedia ports, and gnustep related ports have been updated. 

I've also currently running package builds for i386 and amd64. 

I just blessed the magus run for i386.  That means there are a new set of packages on the FTP server for i386.  I also put out a new ports index for this release tied to the mport tools sqlite3 database.

I've got a dedicated i386 system for package builds again.  It takes a few days to do a run, but it's better than nothing.  It's a Pentium D 805 w/ 2GB RAM. 

MidnightBSD mports now includes the latest Apache release.  This fixes a DOS attack using range headers.  Users are encouraged to upgrade. 

I've just updated the opera port to 11.51.  This version includes some security fixes.  

MidnightBSD has been lagging some other systems with respect to web browsers.  Tonight, I updated Opera, Flash 10 (linux emulation), Firefox 6 and Thunderbird 6 (linux) ports and the nspluginwrapper port.  It's now possible to play flash video on several browsers in current (again) and access youtube.

You must be running 0.4-CURRENT to use some of these.  

mport perl updated

Perl 5.14.1 is now available in mports.  While I've considered updating base, 5.14.2 is scheduled for release next month.  If 0.4 is not ready by the time the release occurs, I'll update it in base, otherwise it will be included in 0.5.

Base Perl issues 

Users having difficulty upgrading from MidnightBSD 0.3 to 0.4 with failures in perl should try freshly checking out perl and building it outside of the full buildworld cycle (cd /usr/src/usr.bin/perl && make && make install) first. 

There's an apache exploit that affects everyone running 2.x.  On MidnightBSD, we only ship 2.2.x now.  A security patch was supposed to be out for this yesterday, but it hasn't happened.  As such, I'm providing a link to the discussion. There has been active discussion on the Apache mailing lists starting on the 24th.  

I've been working hard on getting the port upgrades in place for the next MidnightBSD release.  The following ports have been updated recently:

QT 4.7.3

GNUstep (make, base, back, gui, ...) .  This uses the system compiler now as there are some issues with the GCC 4.4 snap Objective-C support with it.

GCC 4.4.7 snap 6/28

 One of the nice features of Mac OS X is Apple's spotlight.  It makes it easy to find documents because it supports full text search and is aware of different file types.  In the open source world, there are many search tools for Linux, but they all fail in different ways.  Some of them are slow.  Others don't support full text search and rely on inotify.

Linux solutions 

With inotify, the Linux kernel can notify a program that a file has changed by path name.  In the BSD community, we have kqueue that will report changes via fd.  Ideally, one would create a system daemon that can monitor changes in files and update the index on the fly.  This is planned for a future version of msearch(1).  A flaw with most BSD approaches is that it's easy to hit the kern.maxfiles limit as one has to have many directories and files open to detect changes.  kqueue approaches tend to work with UFS and UFS2 file systems only.  Someone using ZFS or fat32 would not get changes unless polling was used. Most modern Linux systems use gamin or FAM to monitor file changes.  

Many of the Linux solutions are under the GPL license. They were not designed for BSD.  I've started down the path of solving this problem.  The first iteration of my work is called msearch.  msearch(1) is a command line tool to search for files on the computer either matching elements of the path or by using the full text search feature.  

Indexing

All text files on the computer can be indexed by msearch.  It uses libmagic to determine the mime type of the file.  This allows it to skip files that are empty, binary, or otherwise useless to the search tool.  

msearch(1) uses two index files generated by a program called msearch.index.  /var/db/msearch.db is a sqlite database containing path information, owner, group, and file size at the time of indexing. /var/db/msearch_full.db contains a sqlite 3 FTS4 full text index of the text files on the computer.  It makes use of zlib to compress the text data.  On my computer, approximately 350,000 files were indexed and 84,000 were considered text files indexable by the full text engine.  Prior to adding compression, the database used 850MB of space. After compression, the file uses 413MB.  Another compression algorithm might cut off additional space at the expense of indexing performance. 

The current version of msearch relies on a periodic script similar to locate(1).  It is run weekly and most be turned on with weekly_msearch_enable="YES" in periodic.conf.  I would like to replace this process with a daemon that handles search requests and indexing.  Apple's search features work in this manner.

Graphical Search

Most of the logic for msearch(1) was placed in a shared library, libmsearch, which can be used to create a graphical search tool.   I envision a sherlock like search tool for the initial release and possibly an integrated solution if MidnightBSD ever gets it's own window manager.  

Security

There are several possible issues with generating an index of all files.  If the index is readable by any user, it could allow one to open the sqlite file and read the contents of sensitive files.  For this reason, I've limited the indexer so that it cannot run as the root user.  Files most be readable by nobody (if using the periodic script) to become part of the index.  

There is also the possibility of sql injection.  The database files aren't writable by normal users and the indexer uses prepare statements.  As the searching functionality is currently using a custom built search string, this could result in undesired behavior.  It's also not recommended to do a search as the root user.  sqlite does have the ability to load extensions, and this feature is used to compress and rank full text data.  The extension loading is turned off right after the database is created to avoid problems form uesrs. 

Future directions

I have a large list of features to add to mserach(1).  I plan to add filtering based on file size, user id, group id, created and modified times. I've considered adding a network search feature in combination with the plans for the search daemon and indexing in near 'real time" with file monitoring.  In order for this to work efficiently, a new kernel interface would need to be created or kqueue would need to be modified.  

I don't intend for this tool to replace locate(1), find(1) or similar search functions, but merely allow users to have an additional option with full text.  

Performance

Full text searches are quire fast.  Simple queries such as searching for Linux are done in seconds.  A search against path names takes longer than locate(1), but is still respectable. locate(1) uses a path compression technique to keep the database small and was optimized for low resources.  msearch(1) takes advantage of the convenience of sqlite 3 and the modern performance of PCs. 

I've finally completed the libxul port (mozilla gecko).  This allows us to build yelp again for Gnome.  I've also created a linux-firefox5 port and a linux-thunderbird5 port.

Opera, curl, tnftp, and tnftpd mports were updated tonight.  I also removed ftp/mirror as the upstream disappeared and the license limited what we could do with the software.