MidnightBSD Developer Journal

There are quite a few new developments with the project.  Archite is experimenting with some very interesting patches.  I'm hoping they'll work out as the enhanced security would be nice. 

ctriv has a good start on the changes to mports.  A bsd.mport.mk file was created which will eventually replace bsd.port.mk.  He's working on several new targets, changing existing targets and has prototypes of a new database format for packages and some perl prototype scripts.  We will most likely implement the final system in C, but perl is an excellent language to prototype with. 

raven has been working on a prototype package management GUI application.  (GNUstep)

I've been very busy the last week and haven't done much.  I did update Enhanced Speedstep (ESS) support with cpufreq to support several Pentium M chips.

 I also increased the default loopback MTU (LOMTU is now 32768).  In some cases, the speed was slightly less but in others it made a significant improvement.   NetBSD has been using a similar value since 4.4 lite and OpenBSD also uses it.  FreeBSD and DragonFly use 16384.  I believe some adjustments to the network stack could improve throughput on lo0 with the new value.

Archite also made adjustments to the umask default to make installing mports easier.  He also added a .kshrc file to the system. 
I've had some feedback on the new website design.  Everyone I've talked to has found one element about it strange.  Surprisingly, its not the new "logo" with the cat eyes.  Most people dislike the gray background or the layout.  We're considering some changes to the design and changing the layout of content.  I designed the current site with a little feedback from archite. 

wintellect has been working with ctriv on the mports overhaul.  ctriv has almost completed the x11 ports transition to /usr/local.  We need to do more testing. 

I've also had some requests to change the CVSup server to add CVSROOT and some other modules.  I'll try to get to that soon.  I'm no expert with CVSup  (just pretend the registered trademark is there) .  I'd appreciate if there are any other changes or addition to CVSup, rsync or the ftp layout that it would be shared with me soon.  I'd rather be coding. 

I was talking with archite today about the DragonFly 1.0 RC1 announcement on slashdot from a few years back.  We talked about the struggle that project went through in the community for forking as well as the amount of work they accomplished going into 1.0 Release. We realized that we don't have a very structured list of goals and a timeframe as that project did from the beginning.  It was rather nice looking back.  Several of us plan to write up some goals in our respective areas of interest and hopefully publish that on midnightbsd.org soon.  I made up a list of changes for userland a few weeks ago that I'll be posting as well.  The DragonFly project focused heavily on the kernel design early on.  In our case, we are more concerned with getting a basic desktop environment working.  That means immediate goals include creating a live cd based installation system with a new installer, overhauling mports to improve package management  (including GUI tools), working on improving management of services as well as providing graphical system preferences, etc.  At the same time, I have a personal wish list which includes doing a lot of work on the network stack, improving file system interopability, adding a journalized file system, possibly trying to get gjournal working, and there's even been talk about eventually bringing zfs in.  We'd like to improve the wiereless user experience including new drivers, easier administration, and so on.  I'd also like to overhaul syscons at some point. 

Then we have the normal userland related changes we'd like to do:
update gcc to something more recent (3.4.6 and later 4.1.x)
binutiles
bind 9.4
sendmail (no i don't plan on switching to postfix)

merge some of the changed to dhclient from OpenBSD and FreeBSD

less

OK, i forget the rest right now.  rc.d cleanup is also planned. 

We've got a new MidnightBSD HTTP mirror.  The Eastern Michigan University Computer Science department is now mirroring our FTP server via HTTP.  Updates are not as fast with this mirror.

You can now download a qemu disk image with MidnightBSD pre-installed. 

I just published a new website design I had been working on.  I haven't had time to polish it, but it was still significantly better than the old design.  This one is not a ripoff of another BSD project.  The new "logo" uses an actual photo of Midnight's eyes.  I thought it was a little creepy.

BtichX is now in mports.  I think this will make several of us happy. Gnome support does not work since we don't have gnome in our tree. 

We're still talking over the proposal for transitioning mports to another system.  It looks like we will probably stay with our own ports tree.  Aside from MirBSD ports, we also talked about pkgsrc.  The conclusion so far is that we need our own ports to maintain the control for tight desktop integration that we'd need.  We didn't see any particular deficiencies in the other projects, just the fact that we don't feel we could get the user experience we are working for. 

In my blog, I will probably continue to call it mbsd but for official channels I guess we'll start calling it Midnight for short.  Someone suggested MNBSD but i don't like that much. 

Our first bug report is in so it appears bugzilla is working now.  http://bugreport.midnightbsd.org

bugreport.midnightbsd.org was fixed.  Apparently I can't type in URLs properly.  Users should now be able to login.  bugzilla stores the base url in the database and this was entered incorrectly. 

linux-firefox needs to be updated to 2.0.0.3 which I will hopefully get to tonight. 

sparc64 release building is broken.  It appears sunlabel is broken so if you upgrade from an older release keep an old copy of this lying around until its fixed.  I was going to add a bug for this when I noticed bugzilla :)

x11-tookits/vte was reported broken today.  I'll be looking at that soon.  The transition is continuing to keep ports out of /usr/X11R6. 

We've been asked about migrating/merging to another ports system which may or may not happen.  We're talking it over right now.  In the mean time, we are operating on the assumption we'll keep our own ports.  There are pros and cons to this approach such as not worrying about some of the ports issues.  We would still need to build guis, package management and other tools on that system and it may or may not hold us back with GNUstep related elements.  The last point is my largest concern at the moment. 

Something to think about is ipfw.  Apple's working on some new extensions to ipfw for 10.5 that might be interesting to do in mbsd (see below lol)  Dynamic rulesets that alter rules based on conditions so that say large storms of traffic to port 80 that are "unusual" would be blocked by 10.5 server, etc.  The concept isn't amazing but their implementation might be.

And now for some good news.  Allbsd.org will start mirroring our CVS and possibly our FTP server.  I'll post details on the midnightbsd website later.

Lastly, I guess the MirBSD project doesn't like us using mbsd as an abbreviation.  I was under the impression they were just using mir or else I wouldn't have gone there.  I suppose some people might be confused so I'd love suggestions on alternatives.  mibsd seems weak.  I guess we could just call it Midnight as DragonFly does with their stuff....  ideas welcome.

I just uploaded an AMD64 snap to the FTP server. It includes all the recent enhancements such as OpenSSH 4.6, new sound code, etc. It does not include binary packages. The mports collection included was taken during transition with xorg. It would be useful to update mports after installing this snap.

The sparc64 snap is not ready yet. There are still some problems with sparc compatibility.

An informal count of ports shows 1077. I guess that means we have 1/16 of the freebsd ports :)

I added spam assassin tonight.

The patch was accepted upstream.

I've had a few complaints about including Opera(c) with MidnightBSD. Some people feel closed source software does not belong in open systems. I would love some feedback on what you think. (comment on this entry or use the forum)

A few changes in the tree have broken AMD64 and sparc64. i386 should be fine. I'm working on it. It was discovered when I was trying to make new snaps.

ctriv is working on migrating X11 ports to install in /usr/local/ instead of /usr/X11R6. This will allow us to migrate to xorg 7.x in the future. Some ports might be broken for a few days while he does the transition.

MidnightBSD now has a bug reporting and tracking system setup. Please add future bug reports to this system.

http://bugreport.midnightbsd.org/

i just created a quick diff for MBSD with tcsh 6.15 which was released very recently.

http://www.midnightbsd.org/tcsh.diff

I also sent this upstream.

I noticed when mports was started after our "ports" issue, the config.sub and config.guess files are lying and saying we are freebsd.  While this might be necessary in ports for now, we should post patched versions for MBSD.  I've had some on my system for some time.  I'll try to do this later.  I also have a small qemu disk image with the latest snap + x11 installed to post on the ftp later.  I was able to install MBSD with the xorg packages on the new ISOs and that's what the qemu image is.  I'm starting to think I need to make my own patches directory that's publicly accessible. 

I am hoping to import tcsh 6.15 soon.  There is at least one known bug with the new version.  It has improved memory allocation/cleanup code though. 

I now have a redistribution agreement with Opera Software to distribute the Opera(c) browser with MidnightBSD.  This is relevant as I'm able to distribute it on ISOs. 

This morning I uploaded a new snapshot without ISOs.  (dated the 18th)  Our make release script is not including the packages properly.  I manually made ISOs with packages for testing.  Its in the snapshots/i386/0.1-Special folder on the FTP server.  In theory, xorg should be installable from the disk1 CD as well as many other packages.  There are about 500MB of packages between disk1 and disk2.  Only fetch disk2 if you intend to use the packages on it.  Mostly its GNUstep, WindowMaker and KDE packages.

I have not had an opportunity to test this ISO yet.  It may not work or not be bootable for some reason. 

As for the release script, I have an idea why it might be failing and can setup a work around for it.  Once these little details are worked out, I will start working on modifying sysinstall to drop a GNUstep base + some apps with installation.  I will also create a metaport for GNUstep related ports and another for MidnightBSD "gui" customizations which will actually setup the graphical environment.  By not including it in base, I'm able to possibly offer non gui installation for servers.  I'm not targeting servers, but I personally use MBSD on servers so its helpful to me.  There are also a few in the project that aren't big GUI fans like our security officer. :)

New ports:
print/gv
net/netatalk
java/mysql-connector-java
devel/log4j
java/jboss4
graphics/quickshow
graphics/gozer
graphics/goom
x11-wm/fvwm2

Updated ports:
shells/zsh
textproc/docproj
x11-tookits/qt33 (note not the latest version)
sysutils/cdrtools
mports/audio/libogg
mports/audio/linux-libogg
x11-tookits/gtk20
lang/php5-extensions
net/samba3
emulators/wine
x11/xscreensaver-kde


Also, archite has commited new sound drivers which improves compatibility with some newer sound cards.  He was inspired by a new acer laptop he acquired.

We haven't shipped anything with Étoilé yet, but in case you are curious they've got a live CD.

http://xdev.org/etoile/

I just posted a freshly generated index on the webserver. make fetchindex grabs this file and its also retrieved when the local index is broken at times. I'm quite bad about keeping it updated as it had be about 45 days since I updated it last. On the upside, the index generated fine the first time.

Its cleanup day for mports. I'm going to attempt to build packages this weekend and do our first package filled isos. That also means you can expect a new snap this weekend if all goes according to plan. That snap will include the timezone changes, the new BSD licensed gzip import from NetBSD/FreeBSD, the libarchive updates, openssh 4.6p1, and so on.

I should also clarify that there is one commited mport that does not work. Mozilla was imported to help us work on the problems with it. Its currently building on some systems, but the GUI does not seem to be connected to the backend code so it does not do much of anything. We suspect a project with their security manager. Adding a new OS to it isn't fun. The linux versions of Mozilla branded browsers are available as well as opera. I may try to get permission from opera to distribute opera on our isos. Its working very well on MBSD. I realize a lot of people like that browser in spite of its closed source and "unique" UI. I have to admit it works very well on unix like systems. I would probably buy it for my wii if i could get a wii!

I just did a test install on an old HP Pavillion made around 2000. It shipped with windows me and has a celeron 700Mhz processor. It took about 5 minutes to install mbsd on it. I'm going to build a few packages on it since its got a clean install.

Archite is continuing to work on building new packages and trying to figure out what is wrong with mozilla. He's also researching making pf our default firewall. I personally like ipfw, but most others I've talked to prefer pf. There are several advantages to pf so we'll see where that goes. The version in our tree is from around OpenBSD 3.7 or so. We'd need to work on updating it.

I also did a successful test package-split today. We need to build a few more packages as we've updated freetype2 and a few other things that mess with a lot of x11 ports.

As far as new ports are concerned, I am simply going through ports I started to work on previously and commiting them if they are now working. Most of them just needed some dependancy which is now in and I forgot about them. Most notably I got several of the broken KDE ports in.

On the topic of GNUstep, we're hoping to start installing GNUstep + xorg as part of the installer soon. If we ship a working snap with packages this weekend, we'll start working on the installer to automate that step. We'll also add some new meta ports to install GNUstep based software. After these steps are completed, we'll do our first release.

Concurrently, ctriv will be working on what we are currenly calling "mport" which is a package management system. pkg_add and other package tools will become wrappers for this new system. It will be written in C and eventually a graphical version will be built as well. Most likely the gui version will be a GNUstep app. He's almost done planning and will be developing a perl based prototype in the near future. It hit me the other day that GNUstep has an installer available. I want to look at that before we go to much farther with "mport" but i suspect some integration will be required.

We're looking at having a database containing all the packages available, their security status, some meta tags describing them, and a few other pieces of information. End users would just do something like mport install pkgname. There would also be mport status, mport list and so on. The hardest step so far has been finding a name for this thing that is not a usability nightmare and still not taken. mport, mpkg and several other variants are taken.. i'm starting to see why gentoo called thier stuff emerge/portage. All the good names were taken.

I'll create an entry with new ports later.

This is an interesting read. The patch is very small.
http://secunia.com/advisories/24490/

Index: sys/kern/uipc_mbuf2.c
===================================================================
RCS file: /cvs/src/sys/kern/uipc_mbuf2.c,v
retrieving revision 1.24
retrieving revision 1.24.2.1
diff -u -p -r1.24 -r1.24.2.1
--- sys/kern/uipc_mbuf2.c 17 Mar 2006 04:15:51 -0000 1.24
+++ sys/kern/uipc_mbuf2.c 7 Mar 2007 19:21:48 -0000 1.24.2.1
@@ -1,4 +1,4 @@
-/* $OpenBSD: uipc_mbuf2.c,v 1.24 2006/03/17 04:15:51 brad Exp $ */
+/* $OpenBSD: uipc_mbuf2.c,v 1.24.2.1 2007/03/07 19:21:48 deraadt Exp $ */
/* $KAME: uipc_mbuf2.c,v 1.29 2001/02/14 13:42:10 itojun Exp $ */
/* $NetBSD: uipc_mbuf.c,v 1.40 1999/04/01 00:23:25 thorpej Exp $ */

@@ -226,16 +226,14 @@ m_dup1(struct mbuf *m, int off, int len,
{
struct mbuf *n;
int l;
- int copyhdr;

if (len > MCLBYTES)
return (NULL);
if (off == 0 && (m->m_flags & M_PKTHDR) != 0) {
- copyhdr = 1;
MGETHDR(n, wait, m->m_type);
+ M_DUP_PKTHDR(n, m);
l = MHLEN;
} else {
- copyhdr = 0;
MGET(n, wait, m->m_type);
l = MLEN;
}
@@ -249,8 +247,6 @@ m_dup1(struct mbuf *m, int off, int len,
if (!n)
return (NULL);

- if (copyhdr)
- M_DUP_PKTHDR(n, m);
m_copydata(m, off, len, mtod(n, caddr_t));
n->m_len = len;


This isn't the first time they've had a problem though:
http://secunia.com/advisories/10801/

I reviewed MidnightBSD code. Our handling is a bit different.

static struct mbuf *
m_dup1(struct mbuf *m, int off, int len, int wait)
{
struct mbuf *n;
int copyhdr;

if (len > MCLBYTES)
return NULL;

if (off == 0 && (m->m_flags & M_PKTHDR) != 0)
copyhdr = 1;
else
copyhdr = 0;

if (len >= MINCLSIZE) {
if (copyhdr == 1)
n = m_getcl(wait, m->m_type, M_PKTHDR);
else
n = m_getcl(wait, m->m_type, 0);
} else {
if (copyhdr == 1)
n = m_gethdr(wait, m->m_type);
else
n = m_get(wait, m->m_type);
}

if (!n)
return NULL; /* ENOBUFS */

if (copyhdr && !m_dup_pkthdr(n, m, wait)) {
m_free(n);
return NULL;
}
m_copydata(m, off, len, mtod(n, caddr_t));
n->m_len = len;
return n;
}

I'm in the middle of updating libarchive + bsdtar in src.  I seemed to get it to get through buildworld + installworld on one system and manually compiling libarchive + tar on another.  I am still a little suspect of it though.  Be sure to test tar thoroughly if you update source and any compilation issues should be reported to luke@midnightbsd.org.   I suppose you could comment on this blog if you have a JJ account as well. 

I've been meaning to setup bugzilla, and I think this might be a good case for why I should have done it sooner. 

As for a RELEASE, I know I promised one several times in the past.  We just don't feel its a good idea to release until we have a simple desktop working.  Perhaps if archite is successful with his mozilla patches and ctriv gets his prototyping done for our new package management system we'll be ready to work on a release.  We don't need the new package system done, but I certainly want to have it planned out before the release.  I do hope wintellect gets some free time to help us prototype it.  He's just getting enough time for bsdnexus.com I think. 

I received an e-mail yesterday from an upset individual who just realized we exist.  He was quite upset about our fork and asked why we forked.  He also assumed that I must have been involved in a fight with the core team of FreeBSD to work as DF and OpenBSD started.   Some of his concerns were understandable, but I don't feel like my actions are similar to the Linux distro scene.  In my view, PC-BSD and DesktopBSD are distros of FreeBSD.  Granted, DesktopBSD flatly states they are not a distro but rather a project writing tools.  I don't see the difference between that and say Gentoo writing portage (tools) for an existing kernel + userland.  Regardless, my point is not to define DesktopBSD's place in the world.  I did not follow the Freesbie, DesktopBSD, PC-BSD path of making a livecd or distro* because I feel that certain changes will need to be made to make MidnightBSD more friendly as a desktop environment.  FreeBSD is "the power to serve" after all.  They also have made close deals with Intel and binary blobs.  I'd rather have an open desktop.  I've said before that I am not against binary blobs in the way that OpenBSD is but I do prefer to avoid them.  FreeBSD has probably hurt the rest of the BSD community by agreeing to binary blobs in base from Intel.  There is an example of a problem with FreeBSD.  I've previously ranted about extreme SMP, treatment of commiters, and other problems with their project.  Some of those issues are being addressed.  I was never a commiter to any other BSD project.  I did submit patches to FreeBSD commiters and I did try to get involved with an HFS+ port to FreeBSD 6.  The former resulted in my code used without credit and the latter was a wasted three months when my hard drive failed and I didn't make a backup like an idiot.  (There are backups of the MBSD cvs, although I could use a tape drive or something...)  I don't feel like defending myself yet again.  People still bring up OpenBSD and DragonFly so I know this won't end regardless of what I do.  I got news for FreeBSD fans... they based theirs on 386BSD!  

I do want to address one concern he had.  "The community is stagnating".  Well, yes it is to some degree.  More likely people are joining younger BSD projects who can't get into FreeBSD or don't have the years of ass kissing to get commit bits.  From my experiences, OpenBSD seems to be much better on this front.  There is still a power structure which is inevitable in all open source projects, but people listen and discuss things.  Its not a flame war.  If I were to go back in time and not start MidnightBSD, I would probably try to join OpenBSD or DragonFly.  They are great projects with some interesting goals.  There is also nothing stopping someone from working on more than one BSD project.  In fact, there are developers doing that with DF, FBSD, etc. 

So why did I fork?  Things weren't getting done on the desktop.  At the end of the day, PC-BSD, DesktopBSD and Freesbie are just going to be ports.  Besides some people don't like KDE.