pandora-build example – Getting Started with pandora-build

May 13th, 2010

Monty Taylor one of the primary contributors to the drizzle project has done some great work putting together a automake/autoconf framework which has been released as pandora-build The framework came about as a result of trying to maintain and improve the build system for several projects mostly related to drizzle or at least belonging to the drizzle developers. It was a pain to duplicate the work in multiple places and it was obvious the setup would be of great use to other projects and so it was pulled out, cleaned up, and released as a skeletal project. One of the best “features” of the project is that it turns on as much -W* and related flags as possible thus ensuring your code is as clean, portable, and bug-free as the compiler is capable of. It can take some getting used to, but trust me this is a very good thing.

If you go and grab the tarball you’ll have a pretty good start on a build system for you project, but as it stands there’s essentially no documentation on what to do next. Given that I’m intending to write a quick-start guide to getting a project off the ground with pandora-build. This post will also give you a starts on working with dynamically linked libraries and executables that depend on them. We’ll also use build out a basic unit test framework using cppunit.

Before we get started I wanted lament the lack of decent autotools documentation on the internet and to mention a good book/source of information that I grabbed several years ago: GNU Autoconf, Automake, and Libtool. There’s another book on the subject that is not yet available (as of this writing May/2010,) but it looks promising: GNU Autoconf, Automake, and Libtool

Ok, with that taken care of the first you’ll want to do is to make sure you have a few dependencies satisfied, a rough list of which follows:

  • g++/make/etc.
  • automake
  • autoconf
  • autoconf-archive
  • libtool
  • python
  • libcppunit (devel libs)

On ubuntu you can run the following to install them:

  sudo apt-get install build-essential automake autoconf autoconf-archive libtool \
    python libcppunit-dev

To check out the results of this tutorial download pandora-example.tar.gz.

Once those are out of the way we can move on to downloading a release of pandora-build which can be found on the right-hand side of the pandora-build project page. Next we need to extract the files and rename the directory.

    $ tar xvzf pandora-build-*.tar.gz
    $ mv pandora-build-*/ pandora-example/
    $ cd pandora-example

We can now look around at the base files. Most of them we’ll leave as-is, but there’s a few changes we’ll need to make. We’ll first look at configure.ac and modify the header and AC_INIT information to suit our project. After that we’ll add a AM_PATH_CPPUNIT line just before AC_CONFIG_FILES and add our pkg-config file to AC_CONFIG_FILES directive. Finall we’ll add some parameters to PANDORA_CANONICAL_TARGET to indicate that we want warnings, that we require cxx, and that we want to skip the shared library visibility checks and support, which you actually shouldn’t do, but it complicates the example. (see http://gcc.gnu.org/wiki/Visibility for information about visibility and look for a future post on the subject.) Anyway, after making these changes configure.ac should look like the following:

# pandora-example
AC_DEFUN([PANDORA_EXAMPLE_VERSION],[0.01])

AC_INIT([pandora-example],PANDORA_EXAMPLE_VERSION,
        [http://something.pandora-example.com])
AC_CONFIG_SRCDIR([m4/pandora_canonical.m4])
AC_CONFIG_AUX_DIR(config)

PANDORA_CANONICAL_TARGET(warnings-always-on, require-cxx, skip-visibility)

AM_PATH_CPPUNIT(1.9.6)

AC_CONFIG_FILES(Makefile helloworld/helloworld.pc)

AC_OUTPUT

We now need to make a few modifications to Makefile.am, namely removing most of it’s contents leaving only ACLOCAL_AMFLAGS. Following that we’ll place a few variable declarations up top and add a couple include directives. We’ll end up with the following Makefile.am:

# pandora-example

ACLOCAL_AMFLAGS= -I m4

# includes append to these:
SUFFIXES =
TESTS =
check_PROGRAMS =
noinst_HEADERS =
nobase_nodist_include_HEADERS =
nobase_dist_include_HEADERS =
bin_PROGRAMS =
sbin_PROGRAMS =
lib_LTLIBRARIES =
noinst_LTLIBRARIES =
noinst_PROGRAMS =

include helloworld/include.am
include tests/include.am

We won’t be making use of all of those variables now, but it doesn’t hurt anything to have them in there so that if/when they’re necessary we don’t have to go back and add them. If you prefer feel free to remove everything we’re not immediately using.

Ok, that’s it for the editing portion of things, now we’ll need to create some directories and files. We’ll start by building the directory structure:

    $ mkdir -p helloworld/include/helloworld-1.0 tests

And then continue on to create some files in those directories starting with helloworld/include.am:

# vim:ft=automake

# library

INCNAME=helloworld-1.0

lib_LTLIBRARIES+=helloworld/libhelloworld.la

helloworld_libhelloworld_ladir=$(includedir)/$(INCNAME)
helloworld_libhelloworld_la_CPPFLAGS=-I. -I$(srcdir)/helloworld/include
helloworld_libhelloworld_la_LDFLAGS=
helloworld_libhelloworld_la_HEADERS=\
			 helloworld/include/$(INCNAME)/phrase.h
helloworld_libhelloworld_la_SOURCES=\
			 helloworld/phrase.cpp

pkgconfigdir= $(libdir)/pkgconfig
pkgconfig_DATA= helloworld/helloworld.pc

# exec

helloworld_helloworld_CPPFLAGS=-I. -I$(srcdir)/helloworld/include
helloworld_helloworld_LDFLAGS=
helloworld_helloworld_LDADD=helloworld/libhelloworld.la
helloworld_helloworld_SOURCES=\
			      helloworld/helloworld.cpp

bin_PROGRAMS+=helloworld/helloworld

There’s a couple things going on there as we’re defining both a shared library and an executable that depends on it. The first variable, INCNAME, is just a helper that prevents us from having to duplicate a value in multiple places. The lib_LTLIBRARIES line adds a new shared library to the set of things we’re going to build and the next set of lines tell automake about the shared library. I don’t want to get too far in to what’s going on here or else I’ll end up (re-)writing a book, but I do want to point out the use of named and versioned include directories as it’s a useful best-practice. In this case helloworld’s headers will be placed in the directory $(includedir)/helloworld-1.0/ and thus will be included by apps that need them with directives like “#include <helloworld-1.0/parser.h>.” This allows multiple major versions of your library to co-exist on a single system. The final piece of the library section tells automake where to find our pkg-config file and where to install it (more later.)

Ok the exec section is more straightforward, to anyone familiar with automake anyway, and just defines an executable that depends on libhelloworld.so that we defined above and makes sure that the app can get at the headers when it is compiling (before they’re installed or more correctly make sure that it uses the source versions, not installed ones.) The final line bin_PROGRAMS adds our exec to the list of executables that automake should build and install.

Now we’ll move on to the include.am file in the tests directory. It’s similar to the one we just created, but doesn’t create (shared) libraries or installed executables. It instead creates check_PROGRAMS, executables that are built and run with make check.

# vim:ft=automake

TESTS+=\
      tests/phrase_test

check_PROGRAMS+=$(TESTS)

tests_phrase_test_CPPFLAGS=$(CPPUNIT_CFLAGS) -I$(srcdir)/helloworld/include
tests_phrase_test_LDFLAGS=$(CPPUNIT_LIBS) helloworld/libhelloworld.la
tests_phrase_test_SOURCES=tests/phrase_test.cpp

We have one more “infrastructure” file to deal with for now which will add pkg-config functionality to our shared library. The following needs to be placed in helloworld/helloworld.pc.in

prefix=@prefix@
exec_prefix=@exec_prefix@
libdir=@libdir@
includedir=@includedir@

Name: helloworld
Description: Hello World Pandora Build Example
Version: @VERSION@
Libs: -L${libdir} -lhelloworld
Cflags: -I${includedir}

autoconf/configure will take the above file and create helloworld.pc substituting all of the actual values in place of their variables and automake will install it to the appropriate directory thanks to the lines we added to helloworld/include.am’s library section.

That leaves us with the code. We won’t spend too much time on it as that’s not the purpose of the post. I’ll just tell you the file name and provide the contents to place in it. Here we go.

helloworld/include/helloworld-1.0/phrase.h:

#ifndef _HELLOWORLD_1_0_PHRASE_H_
#define _HELLOWORLD_1_0_PHRASE_H_

#include <string>

namespace helloworld
{
    class Phrase
    {
    private:
        std::string _text;

    public:
        Phrase (const std::string & text);
        virtual ~Phrase ();

        const std::string & getText() const
        {
            return this->_text;
        }

        void setText(const std::string & text)
        {
            this->_text = text;
        }

    protected:
        Phrase (const Phrase & rhs);

        Phrase & operator= (const Phrase & rhs);

    };
}

#endif // _HELLOWORLD_1_0_PHRASE_H_

helloworld/phrase.cpp:

#include "config.h"

#include "helloworld-1.0/phrase.h"

using namespace helloworld;

Phrase::Phrase (const std::string & text) : _text (text)
{
}

Phrase::~Phrase ()
{
}

Phrase::Phrase (const Phrase & rhs) : _text (rhs._text)
{
}

Phrase & Phrase::operator= (const Phrase & rhs)
{
    this->_text = rhs._text;

    return *this;
}

helloworld/helloworld.cpp:

#include "config.h"

#include <stdio.h>

#include "helloworld-1.0/phrase.h"

int main (int argc, char ** argv)
{
    (void)argc;
    (void)argv;

    helloworld::Phrase phrase ("Hello World!");

    fprintf (stdout, "%s\n", phrase.getText().c_str());

    return 0;
}

tests/phrase_test.cpp:

#include <cppunit/BriefTestProgressListener.h>
#include <cppunit/CompilerOutputter.h>
#include <cppunit/extensions/HelperMacros.h>
#include <cppunit/extensions/TestFactoryRegistry.h>
#include <cppunit/TestResultCollector.h>
#include <cppunit/TestRunner.h>
#include <cppunit/TestResult.h>
#include <cppunit/ui/text/TestRunner.h>

#include <helloworld-1.0/phrase.h>

class PhraseTest : public CPPUNIT_NS::TestFixture
{
    CPPUNIT_TEST_SUITE (PhraseTest);
    CPPUNIT_TEST (testPhrase);
    CPPUNIT_TEST_SUITE_END ();

public:
    void setUp ();
    void tearDown ();

    void testPhrase ();
};

CPPUNIT_TEST_SUITE_REGISTRATION (PhraseTest);

void PhraseTest::setUp ()
{
}

void PhraseTest::tearDown ()
{
}

void PhraseTest::testPhrase ()
{
    std::string str = "Hello World!";
    helloworld::Phrase phrase (str);
    CPPUNIT_ASSERT_EQUAL (str, phrase.getText ());
    CPPUNIT_ASSERT ("" != phrase.getText ());

    std::string empty = "";
    phrase.setText (empty);
    CPPUNIT_ASSERT_EQUAL (std::string (empty), phrase.getText ());
    CPPUNIT_ASSERT (str != phrase.getText ());
}

int main (int argc, char ** argv)
{
    (void)argc;
    (void)argv;

    // informs test-listener about testresults
    CPPUNIT_NS::TestResult testresult;

    // register listener for collecting the test-results
    CPPUNIT_NS::TestResultCollector collectedresults;
    testresult.addListener (&collectedresults);

    // register listener for per-test progress output
    CPPUNIT_NS::BriefTestProgressListener progress;
    testresult.addListener (&progress);

    // insert test-suite at test-runner by registry
    CPPUNIT_NS::TestRunner testrunner;
    testrunner.addTest
        (CPPUNIT_NS::TestFactoryRegistry::getRegistry ().makeTest ());
    testrunner.run (testresult);

    // output results in compiler-format, with a custom error format that works
    // with a top-level Makefile.am and include.am's
    CPPUNIT_NS::CompilerOutputter compileroutputter (&collectedresults,
                                                     std::cerr,
                                                     "%p:%l:");
    compileroutputter.write ();

    // return 0 if tests were successful
    return collectedresults.wasSuccessful () ? 0 : 1;
}

Ok that’s it. You can now run autorun.sh, configure, and make and you “should” end up with a working shared library, pkg-config file, and executable. To test it all out do:

    $ ./config/autorun.sh
    $ ./configure --prefix=/tmp/helloworld-test
    $ make
    $ make check
    $ make install

Congrats, you now have a skeletal pandora-build based system. You can go in and add/remove shared libraries, headers, execs, tests, … as needed mostly by copy-n-pasting the above code and stuff you can find in other projects (both libdrizzle and drizzle are good, albeit it complicated, examples.) Feel free to ask questions if you have them and be sure to subscribe as I plan to make this a series of posts on the subject. Some of the planned topics include what needs to be in my repository (.gitignores,) integrating valgrind and other code checking/debugging utilities, logging, distribution/tarring things up, and maybe even building debian and redhat packages.

Rails/ActiveRecord::Base acts_as_obfuscated – hide your object ids

September 27th, 2009

I’ve always liked the idea of obfuscating the ids of my ActiveRecord::Base objects when using them in url. There’s lots of ways to go about this and through past projects I’ve used various mechanisms to accomplish it. Sometimes it’s nice and others it’s required. You might want to hide the number of users/objects/whatevers you have in you system or make it harder for malicious users to gain access to resources by guessing urls.

Well regardless of the reason(s) I’ve created a simple plugin/mixin, acts_as_obfuscated, that does exactly this with a single line of code. An example is in order:

class User < ActiveRecord::Base
  acts_as_obfuscated

  ...
end

and that's it.

$ ./script/console
Loading development environment (Rails 2.3.4)
>> u = User.create(:name => 'Bob')
=> #
>> u.id
=> 4
>> u.eid
=> "diBGnp"
>> User.find(u.id)
=> #
>> User.find(u.eid)
=> #
>>

The piece that's not shown above is an implementation of to_param.

def to_param
    self.eid
end

The effect of this is that anywhere you provider a user object in an 'id => user' param you get the self.eid rather than the default to_param of self.id. So a url that would look like http://mysite.com/users/4 would become http://mysite.com/users/diBGnp.

=link_to(user.name, :controller => 'users', :action => 'show', :id => user)

<a href="http://mysite.com/users/diBGnp">Bob</a>

acts_as_obfuscated doesn't get int the way of custom to_param functions so long as the first portion is the acts_as_obfuscated to_param function:

class User < ActiveRecord::Base
  acts_as_obfuscated

  def to_param
    CGI.escape("#{super.to_param}-#{self.name}").gsub(/\./, '_')
  end
end

$ ./script/console
Loading development environment (Rails 2.3.4)
u= U>> u= User.last
=> #
>> u.to_param
=> "diBGnp-Bob"
>> User.find(u.to_param)
=> #

That will allow you to have seo/ad placement friendly urls without exposing your internal object identifiers.

Anyway, check it out, use it, let me know what you think. The code can be found on github: http://github.com/ross/acts_as_obfuscated.

It can be installed as a plugin by running the command:

./script/plugin install git://github.com/ross/acts_as_obfuscated.git

If you want to see it in action check out: ClBrow - A Visual way to Shop Craigslist, which is a bit slower than I'd like to do the load on my dreamhost db, but hopefully I'll get around to fixing that soon...

-rm

Armadillo Aerospace – Rocket Tug-O-War

September 13th, 2009

Some context… What do you think NASA would do if it wanted to test the effects of wind gusts on one of its rockets? How much do you think they’d spend designing, planning, and testing the test system? For better or worse this is why space will ultimately be concurred not by government programs, but by private enterprises (granted for the a lot of the private endeavors are being funded by NASA and friends.)

“We wanted to get a feel for how the vehicle would react to sudden changes in wind speeds as it moved upward. The only thing we could think of to induce a similar force was to attach a really long cable to one of the legs and have someone pull on it during a hover.”

via Armadillo Aerospace – News Archive.

YouTube Preview Image

(original video)

301 Redirects – Getting Old Feed URLs to Work With Wordpress

April 21st, 2009

If you have an established blog and have readers who subscribe to your feed you’ll likely loose them when you migrate to Wordpress and your RSS feed URL changes. Wordpress feed look like http://www.mysite.com/blog/feed/ if you’re blog is at http://www.mysite.com/blog/. My previous blog software’s RSS feed url looked like http://www.mysite.com/blog/?wl_mode=rss2. I didn’t have too many subscribers, but there were a few and I didn’t want to leave them hanging so I set about using a 301 permanent redirect to solve this problem.

Fixing this is pretty straightforward, just a couple of lines of code. For me this the following placed near the top of index.php does the trick.

if ($_GET['wl_mode'] == 'rss') {
        header("HTTP/1.1 301 Moved Permanently");
        header("Location: http://" . $_SERVER['SERVER_NAME'] .  "/blog/feed");
        exit();
}

It looks for the get parameter wl_mode to be ‘rss’. If wl_mode is defined and equal to ‘rss’ it sets two headers in the response and then exits. The first tells the client to redirect and that the redirect is permanent. The second gives the location to redirect to, the server name of the request, from the variable so that it matches whatever hostname the request was made to and the path of Wordpress’s RSS feed, ‘/feed’. The ‘/blog’ is where Wordpress is installed on my site, if your root is Wordpress you’d just have ‘/feed’.

What if your old url wasn’t wl_mode=rss. If it’s a different parameter or set of parameters you’d just swap them out. What if the old feed is not a parameter, but a URL/path? Something like the snippet below should be useful there.

if ($_SERVER['REQUEST_URI'] == '/blog/old/feed/path') {
        header("HTTP/1.1 301 Moved Permanently");
        header("Location: http://" . $_SERVER['SERVER_NAME'] .  "/blog/feed");
        exit();
}

5 Minute Custom Wordpress Theme

April 18th, 2009

Creating a custom wordpress theme is really easy especially if you’re starting with a template (a site design you want to use.) This is often the case when you’re trying to add a blog to an existing site or convert an existing website you’re happy with to use wordpress for blogging/content management. I won’t go in to details about how to install wordpress, there’s already a great guide for that. I’ll just outline the steps involved in creating a custom them to get wordpress to look & feel the way you want it to.

Laying the Foundation – Creating the Theme Directory

We’ll start out creating a directory to house our theme files (there’s only going to be 2 of them.) To do that we’ll log on to our server and execute the following:

$ cd /whereever/wordpress/is/installed/wp-content/themes
$ mkdir custom

If you don’t have shell access to you server use whatever mechanism you’ve uploaded/edited the sites files with in the past.

Step One – HTML – index.php

There’s only two files required to create a template for wordpress, index.php and style.css. We’ll start with index.php. To create an initial version of this file we’ll pick up where we left off a minute ago and do the following. If you’re familiar with another editor, feel free to use it, choices include vi, emacs, … but pico is one of the simplest to use (ctrl-O to save, ctrl-X to exit, more commands are listed across the bottom.)

$ cd custom
$ pico index.php

That will start up pico editing a file named index.php, the main template file for you new custom theme. We’ll start with a simple html page you can copy-n-paste in to this file.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" <?php language_attributes(); ?>>
  <head>
    <meta http-equiv="Content-Type" content="<?php bloginfo('html_type'); ?>;
      charset=<?php bloginfo('charset'); ?>" />
    <title>
      <?php wp_title('«', true, 'right'); ?> <?php bloginfo('name'); ?>
    </title>
    <link rel="stylesheet" href="<?php bloginfo('stylesheet_url'); ?>"
      type="text/css" media="screen" />
    <link rel="alternate" type="application/rss+xml"
      title="<?php bloginfo('name'); ?> RSS Feed"
      href="<?php bloginfo('rss2_url'); ?>" />
    <link rel="alternate" type="application/atom+xml"
      title="<?php bloginfo('name'); ?> Atom Feed"
      href="<?php bloginfo('atom_url'); ?>" />
    <link rel="pingback" href="<?php bloginfo('pingback_url'); ?>" />
  </head>
  <body>
    <div id='header'>
      Insert Your Header HTML here
    </div>
    <div id='content' class='span-16 prepend-1'>
        <?php if (have_posts()) : ?>
          <?php while (have_posts()) : the_post(); ?>
            <div <?php post_class() ?> id="post-<?php the_ID(); ?>">
              <h2><a href="<?php the_permalink() ?>" rel="bookmark"
                title="Permanent Link to <?php the_title_attribute();
                ?>"><?php the_title(); ?></a></h2>
              <small>
                <?php the_time('F jS, Y') ?>
                <!-- by <?php the_author() ?> -->
              </small>
              <div class="entry">
                <?php the_content('Read the rest of this entry »'); ?>
              </div>
              <p class="postmetadata"><?php the_tags('Tags: ', ', ', '<br />');
                ?> Posted in <?php the_category(', ') ?> |
                <?php edit_post_link('Edit', '', ' | '); ?>
                <?php comments_popup_link('No Comments »',
                                          '1 Comment »',
                                          '% Comments »'); ?>
              </p>
            </div>
          <?php endwhile; ?>
            <div class="navigation">
              <div class="alignleft"><?php
                next_posts_link('« Older Entries') ?></div>
              <div class="alignright"><?php
                previous_posts_link('Newer Entries »') ?></div>
            </div>
        <?php else : ?>
          <h2 class="center">Not Found</h2>
          <p class="center">Sorry, but you are looking for something that isn't
            here.</p>
          <?php get_search_form(); ?>
        <?php endif; ?>
      </div>
      <div id='sidebar' class='span-6 last'>
        <?php get_sidebar(); ?>
      </div>
    <div id='footer'>
      Insert Your Footer HTML here
    </div>
  </body>
</html>

Don’t worry if that looks like a mass of gibberish, there’s only a small portion of it that you’ll have to worry about, the two sections in red. In them you will place your header and footer HTML, whatever logos and/or text you’d like to see at the top of the page. If you’d like to have a navagation bar across the page you can create a second div following the header dive and put links to the various sections of you site there. The footer is a good place to put a copyright notice, links to email you or any other information you’d like to have appear at the bottom of all of your pages.

If you’re working with an existing template you want to insert wordpress into, you’ll take the section in blue and place it in the content section of your template. You may have to mess around with it a bit to get exactly what you’re looking for, but keep at it it shouldn’t take too long.

Step Two – CSS – style.css

We’re half the way to a new custom Wordpress theme. The next thing we’ll need to do is create style.css.

$ pico style.css

At this point if you want to save the file you can go to the admin section for you blog and click on the ‘Appearance’ link and you should see your new ‘custom’ theme. Clicking on it should pop up a preview of what your blog will look like using this theme. It probably won’t look like much yet, but it’s a nice clean workspace in which you’ll be able to mold things to your liking. If you don’t have visitors to your blog yet, or don’t mind them seeing the work in progress you may go head and apply your new theme. If you’re not ready for that you’ll need to continue to use the preview feature to view your work.

So one of the biggest problems with this theme is the sidebar is way down at the bottom below all of the content. We’ll need to add some css to address this issue, luckily there’s not much to it, at least to move the sidebar up. You’ll just need to add the following to style.css and refresh.

#header
{
}

#content
{
  float: left;
  width: 600px;
}

#sidebar
{
  float: left;
  width: 200px;
}

#footer
{
  clear: both;
}

The css above makes both #content and #sidebar float left and then limits #content’s width to 600 pixels and the sidebar to 200. So the blog will be 800 pixles wide. The only other thing going on is that we’ve asked the footer to clear both, which essentially means that it should go below any floating divs before it. This is obviously pretty rudimentary and doesn’t do much for the ascetic appeal of our blog, but everything “works” from here it’s just fiddling with css (which is way beyond the scope of this post.) Take a look at the HTML generated by this theme using view source and you should be able to track down the id’s and class’s you need to address to shape things up. Web developer Tool-bar can be really helpful for this work, check it out.

Conclusions

So we’ve created a simple, although still ugly, wordpress theme from scratch. It uses lots of defaults that can be customized to your liking, but it’s a good start. If you have any questions feel free to hit me up at rwmcfa1 <at> neces.com. Don’t have the time and/or desire to mess with custom wordpress instalation/development get in touch.

Integrating FreeBSD, ZFS, and Periodic; snapshots and scrubs

April 15th, 2009

ZFS on FreeBSD is powerful, especially when coupled with periodic taking hourly, daily, weekly, monthly, … snapshots. In the following post I’ll provide the scripts & config necessary to customize and walk you step-by-step through setting up zfs snapshots and scrubs with periodic on FreeBSD.

Periodic’s main advantage over the more traditional and obvious method of running a script from a cron job is integration with the notification emails and standard configuration mechanism. That may not sound like much, but that means you a year down the road (or someone else that comes to the system) only has to look in the obvious place to figure out what’s going on or make changes.

This is going to be a long post, but there’s a decent amount of code & config to walk through. The files being discussed have been tar’d up and the latest version of them can be downloaded from here.

Configuration – the stuff you might actually want to muck with

We’ll start with the configuration (/etc/periodic.conf) as it’s the most relevant portion or at least the most likely to be edited. Out of the box FreeBSD supports daily, weekly, and monthly periodic tasks, we’re going to be adding an hourly so that we can do hourly snapshots. The first section of config sets up who output from the hourly script should go to, whether it should be sent if everything succeeded, if something failed, or if something is mis-configured. Hourly emails seem a bit much so we’ve disabled them when everything goes well. We do want messages about errors and I’ve just left badconfig to the same value as all of the other time-frames.

# Hourly options
hourly_output="root"					# user or /file
hourly_show_success="NO"				# scripts returning 0
hourly_show_info="YES"					# scripts returning 1
hourly_show_badconfig="NO"				# scripts returning 2

The next section configures hourly snapshots. In this case we’re enabling them for the pool tank and keeping the 6 most recent around. There are defaults, that we’ll see later, for both pools and keep so the only required value here is enable. To specify more than one pool add them space seperated to the config string, e.g. “tank boat plane”

# 000.zfs-snapshot
hourly_zfs_snapshot_enable="YES"
hourly_zfs_snapshot_pools="tank"
hourly_zfs_snapshot_keep=6

The daily section is almost identical, but we instead keep the last 7 days. We’re also enabling a daily zfs status script that is in the default setup, but disabled.

# Daily options

# 000.zfs-snapshot
daily_zfs_snapshot_enable="YES"
daily_zfs_snapshot_pools="tank"
daily_zfs_snapshot_keep=7

# 404.status-zfs
daily_status_zfs_enable="YES"

Weekly and Monthly have the same configuration options, we’re keeping the last 5 weeks, and last 2 months below.

# Weekly options

# 000.zfs-snapshot
weekly_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_pools="tank"
weekly_zfs_snapshot_keep=5

# Monthly options

# 000.zfs-snapshot
monthly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_pools="tank"
monthly_zfs_snapshot_keep=2

A final section configures the monthly scrubbing. Similarlly to the snapshot config sections there’s an enable line and pools line. Here we’ve enabled the monthly scrub on the tank pool.

# 998.zfs-scrub
monthly_zfs_scrub_enable="YES"
monthly_zfs_scrub_pools="tank"

periodic hourly – adding hourly script support to periodic

The next thing we need to do is add hourly support to periodic. While that might sound complicated it’s actually very straightforward. We start by creating a directory to house the hourly files.

# mkdir /etc/periodic/hourly

And then add the following line to /etc/crontab just before the line for ‘periodic hourly’

1	*	*	*	*	root	periodic hourly

That’s it you now have a place to put scripts that will be run every hour on the :01.

hourly/daily/weekly/monthly scripts – adding hourly script support to periodic

Now we’ll get to the scripts that make all of this configuration do something. It’s unlikely that you’ll ever have to much with any of these, but in case your curious I’ll go ahead and walk though them. We’ll start with the hourly snapshot script (/etc/periodic/hourly/000.zfs-snapshot.)

 1  #!/bin/sh
 2
 3  # If there is a global system configuration file, suck it in.
 4  #
 5  if [ -r /etc/defaults/periodic.conf ]
 6  then
 7      . /etc/defaults/periodic.conf
 8      source_periodic_confs
 9  fi
10
11  pools=$hourly_zfs_snapshot_pools
12  if [ -z "$pools" ]; then
13      pools='tank'
14  fi
15
16  keep=$hourly_zfs_snapshot_keep
17  if [ -z "$keep" ]; then
18      keep=6
19  fi
20
21  case "$hourly_zfs_snapshot_enable" in
22      [Yy][Ee][Ss])
23          . /etc/periodic/zfs-snapshot
24          do_snapshots "$pools" $keep 'hourly'
25          ;;
26      *)
27          ;;
28  esac

Lines 3-9 is boilerplate periodic script stuff. 11-19 look for the values we configured earlier and use defaults if they’re not specified. 21, 22, 25, and 26 are case shell scripting case statement stuff that’s borrowed from one of the other periodic scripts, mainly just makes sure that hourly_zfs_snapshot_enable is set to YES, ignoring case. Line 32 pulls in (think #include) some common zfs snapshotting code that we’ll get to next and finally line 24 calls the snapshotting function for the configured pools, keep count, and the type of hourly. The daily, weekly, and monthly scripts are identical with hourly replaced with the appropriate value throughout.

zfs-snapshot – the workhorse

There’s too much here to walk though in detail so I’ll let you read through the code. I’ve tried to do a decent job of in-line commenting. If you have questions or want clarification feel free to ask…

#!/bin/sh

# checks to see if there's a scrub in progress
scrub_in_progress()
{
  pool=$1

  if zpool status $pool | grep "scrub in progress" > /dev/null; then
    return 0
  else
    return 1
  fi
}

# take the appropriately named snapshot
create_snapshot()
{
    pool=$1

    case "$type" in
        hourly)
        now=`date +"$type-%Y-%m-%d-%H"`
        ;;
        daily)
        now=`date +"$type-%Y-%m-%d"`
        ;;
        weekly)
        now=`date +"$type-%Y-%U"`
        ;;
        monthly)
        now=`date +"$type-%Y-%m"`
        ;;
        *)
        echo "unknown snapshot type: $type"
        exit 1
    esac

    # create the now snapshot
    snapshot="$pool@$now"
    # look for a snapshot with this name
    if zfs list -H -o name | sort | grep "$snapshot$" > /dev/null; then
        echo "	snapshot, $snapshot, already exists"
    else
        echo "	taking snapshot, $snapshot"
        zfs snapshot -r $snapshot
    fi
}

# delete the named snapshot
delete_snapshot()
{
    snapshot=$1
    echo "	destroying old snapshot, $snapshot"
    zfs destroy -r $snapshot
}

# take a type snapshot of pool, keeping keep old ones
do_pool()
{
    pool=$1
    keep=$2
    type=$3

    # create the regex matching the type of snapshots we're currently working
    # on
    case "$type" in
        hourly)
        # hourly-2009-01-01-00
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]$"
        ;;
        daily)
        # daily-2009-01-01
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]$"
        ;;
        weekly)
        # weekly-2009-01
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]"
        ;;
        monthly)
        # monthly-2009-01
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]"
        ;;
        *)
        echo "unknown snapshot type: $type"
        exit 1
    esac

    create_snapshot $pool $type

    # get a list of all of the snapshots of this type sorted alpha, which
    # effectively is increasing date/time
    # (using sort as zfs's sort seems to have bugs)
    snapshots=`zfs list -H -o name | sort | grep $regex`
    # count them
    count=`echo $snapshots | wc -w`
    if [ $count -ge 0 ]; then
        # how many items should we delete
        delete=`expr $count - $keep`
        count=0
        # walk through the snapshots, deleting them until we've trimmed deleted
        for snapshot in $snapshots; do
            if [ $count -ge $delete ]; then
                break
            fi
            delete_snapshot $snapshot
            count=`expr $count + 1`
        done
    fi
}

# take snapshots of type, for pools, keeping keep old ones,
do_snapshots()
{
    pools=$1
    keep=$2
    type=$3

    echo ""
    echo "Doing zfs $type snapshots:"
    for pool in $pools; do
        if scrub_in_progress $pool; then
          echo "	skipping snapshot of $pool, scrub in progress"
        else
          do_pool $pool $keep $type
        fi
    done
}

We’re about 1% human

April 8th, 2009

A few months back I read a book titled A Short History of Nearly Everything. It really enjoyed the book. It was chocked full of interesting history, related people and developments, and factoids. One of the simplest and best was that the human body has about 10x the bacterial cells as human. That’s staggering enough, 1 trillion human cells, 10 trillion bacterial, but a great presentation I just watched on the communication mechanisms of bacterial cells added that there’s about 100 x more genetic code in those bacteria.

The presentation is about the ways in which bacteria communicate. Apparently there is both an intra and inter-species mechanisms and scientists has successfully mimicked the molecules (hormones) involved. The hope is the work will lead to the next generation of antibiotics (a development that is seriously needed.) Check out the talk, one of many great TED talks.