pandora-build example – Getting Started with pandora-build

May 13th, 2010

Monty Taylor one of the primary contributors to the drizzle project has done some great work putting together a automake/autoconf framework which has been released as pandora-build The framework came about as a result of trying to maintain and improve the build system for several projects mostly related to drizzle or at least belonging to the drizzle developers. It was a pain to duplicate the work in multiple places and it was obvious the setup would be of great use to other projects and so it was pulled out, cleaned up, and released as a skeletal project. One of the best “features” of the project is that it turns on as much -W* and related flags as possible thus ensuring your code is as clean, portable, and bug-free as the compiler is capable of. It can take some getting used to, but trust me this is a very good thing.

If you go and grab the tarball you’ll have a pretty good start on a build system for you project, but as it stands there’s essentially no documentation on what to do next. Given that I’m intending to write a quick-start guide to getting a project off the ground with pandora-build. This post will also give you a starts on working with dynamically linked libraries and executables that depend on them. We’ll also use build out a basic unit test framework using cppunit.

Before we get started I wanted lament the lack of decent autotools documentation on the internet and to mention a good book/source of information that I grabbed several years ago: GNU Autoconf, Automake, and Libtool. There’s another book on the subject that is not yet available (as of this writing May/2010,) but it looks promising: GNU Autoconf, Automake, and Libtool

Ok, with that taken care of the first you’ll want to do is to make sure you have a few dependencies satisfied, a rough list of which follows:

  • g++/make/etc.
  • automake
  • autoconf
  • autoconf-archive
  • libtool
  • python
  • libcppunit (devel libs)

On ubuntu you can run the following to install them:

  sudo apt-get install build-essential automake autoconf autoconf-archive libtool \
    python libcppunit-dev

To check out the results of this tutorial download pandora-example.tar.gz.

Once those are out of the way we can move on to downloading a release of pandora-build which can be found on the right-hand side of the pandora-build project page. Next we need to extract the files and rename the directory.

    $ tar xvzf pandora-build-*.tar.gz
    $ mv pandora-build-*/ pandora-example/
    $ cd pandora-example

We can now look around at the base files. Most of them we’ll leave as-is, but there’s a few changes we’ll need to make. We’ll first look at configure.ac and modify the header and AC_INIT information to suit our project. After that we’ll add a AM_PATH_CPPUNIT line just before AC_CONFIG_FILES and add our pkg-config file to AC_CONFIG_FILES directive. Finall we’ll add some parameters to PANDORA_CANONICAL_TARGET to indicate that we want warnings, that we require cxx, and that we want to skip the shared library visibility checks and support, which you actually shouldn’t do, but it complicates the example. (see http://gcc.gnu.org/wiki/Visibility for information about visibility and look for a future post on the subject.) Anyway, after making these changes configure.ac should look like the following:

# pandora-example
AC_DEFUN([PANDORA_EXAMPLE_VERSION],[0.01])

AC_INIT([pandora-example],PANDORA_EXAMPLE_VERSION,
        [http://something.pandora-example.com])
AC_CONFIG_SRCDIR([m4/pandora_canonical.m4])
AC_CONFIG_AUX_DIR(config)

PANDORA_CANONICAL_TARGET(warnings-always-on, require-cxx, skip-visibility)

AM_PATH_CPPUNIT(1.9.6)

AC_CONFIG_FILES(Makefile helloworld/helloworld.pc)

AC_OUTPUT

We now need to make a few modifications to Makefile.am, namely removing most of it’s contents leaving only ACLOCAL_AMFLAGS. Following that we’ll place a few variable declarations up top and add a couple include directives. We’ll end up with the following Makefile.am:

# pandora-example

ACLOCAL_AMFLAGS= -I m4

# includes append to these:
SUFFIXES =
TESTS =
check_PROGRAMS =
noinst_HEADERS =
nobase_nodist_include_HEADERS =
nobase_dist_include_HEADERS =
bin_PROGRAMS =
sbin_PROGRAMS =
lib_LTLIBRARIES =
noinst_LTLIBRARIES =
noinst_PROGRAMS =

include helloworld/include.am
include tests/include.am

We won’t be making use of all of those variables now, but it doesn’t hurt anything to have them in there so that if/when they’re necessary we don’t have to go back and add them. If you prefer feel free to remove everything we’re not immediately using.

Ok, that’s it for the editing portion of things, now we’ll need to create some directories and files. We’ll start by building the directory structure:

    $ mkdir -p helloworld/include/helloworld-1.0 tests

And then continue on to create some files in those directories starting with helloworld/include.am:

# vim:ft=automake

# library

INCNAME=helloworld-1.0

lib_LTLIBRARIES+=helloworld/libhelloworld.la

helloworld_libhelloworld_ladir=$(includedir)/$(INCNAME)
helloworld_libhelloworld_la_CPPFLAGS=-I. -I$(srcdir)/helloworld/include
helloworld_libhelloworld_la_LDFLAGS=
helloworld_libhelloworld_la_HEADERS=\
			 helloworld/include/$(INCNAME)/phrase.h
helloworld_libhelloworld_la_SOURCES=\
			 helloworld/phrase.cpp

pkgconfigdir= $(libdir)/pkgconfig
pkgconfig_DATA= helloworld/helloworld.pc

# exec

helloworld_helloworld_CPPFLAGS=-I. -I$(srcdir)/helloworld/include
helloworld_helloworld_LDFLAGS=
helloworld_helloworld_LDADD=helloworld/libhelloworld.la
helloworld_helloworld_SOURCES=\
			      helloworld/helloworld.cpp

bin_PROGRAMS+=helloworld/helloworld

There’s a couple things going on there as we’re defining both a shared library and an executable that depends on it. The first variable, INCNAME, is just a helper that prevents us from having to duplicate a value in multiple places. The lib_LTLIBRARIES line adds a new shared library to the set of things we’re going to build and the next set of lines tell automake about the shared library. I don’t want to get too far in to what’s going on here or else I’ll end up (re-)writing a book, but I do want to point out the use of named and versioned include directories as it’s a useful best-practice. In this case helloworld’s headers will be placed in the directory $(includedir)/helloworld-1.0/ and thus will be included by apps that need them with directives like “#include <helloworld-1.0/parser.h>.” This allows multiple major versions of your library to co-exist on a single system. The final piece of the library section tells automake where to find our pkg-config file and where to install it (more later.)

Ok the exec section is more straightforward, to anyone familiar with automake anyway, and just defines an executable that depends on libhelloworld.so that we defined above and makes sure that the app can get at the headers when it is compiling (before they’re installed or more correctly make sure that it uses the source versions, not installed ones.) The final line bin_PROGRAMS adds our exec to the list of executables that automake should build and install.

Now we’ll move on to the include.am file in the tests directory. It’s similar to the one we just created, but doesn’t create (shared) libraries or installed executables. It instead creates check_PROGRAMS, executables that are built and run with make check.

# vim:ft=automake

TESTS+=\
      tests/phrase_test

check_PROGRAMS+=$(TESTS)

tests_phrase_test_CPPFLAGS=$(CPPUNIT_CFLAGS) -I$(srcdir)/helloworld/include
tests_phrase_test_LDFLAGS=$(CPPUNIT_LIBS) helloworld/libhelloworld.la
tests_phrase_test_SOURCES=tests/phrase_test.cpp

We have one more “infrastructure” file to deal with for now which will add pkg-config functionality to our shared library. The following needs to be placed in helloworld/helloworld.pc.in

prefix=@prefix@
exec_prefix=@exec_prefix@
libdir=@libdir@
includedir=@includedir@

Name: helloworld
Description: Hello World Pandora Build Example
Version: @VERSION@
Libs: -L${libdir} -lhelloworld
Cflags: -I${includedir}

autoconf/configure will take the above file and create helloworld.pc substituting all of the actual values in place of their variables and automake will install it to the appropriate directory thanks to the lines we added to helloworld/include.am’s library section.

That leaves us with the code. We won’t spend too much time on it as that’s not the purpose of the post. I’ll just tell you the file name and provide the contents to place in it. Here we go.

helloworld/include/helloworld-1.0/phrase.h:

#ifndef _HELLOWORLD_1_0_PHRASE_H_
#define _HELLOWORLD_1_0_PHRASE_H_

#include <string>

namespace helloworld
{
    class Phrase
    {
    private:
        std::string _text;

    public:
        Phrase (const std::string & text);
        virtual ~Phrase ();

        const std::string & getText() const
        {
            return this->_text;
        }

        void setText(const std::string & text)
        {
            this->_text = text;
        }

    protected:
        Phrase (const Phrase & rhs);

        Phrase & operator= (const Phrase & rhs);

    };
}

#endif // _HELLOWORLD_1_0_PHRASE_H_

helloworld/phrase.cpp:

#include "config.h"

#include "helloworld-1.0/phrase.h"

using namespace helloworld;

Phrase::Phrase (const std::string & text) : _text (text)
{
}

Phrase::~Phrase ()
{
}

Phrase::Phrase (const Phrase & rhs) : _text (rhs._text)
{
}

Phrase & Phrase::operator= (const Phrase & rhs)
{
    this->_text = rhs._text;

    return *this;
}

helloworld/helloworld.cpp:

#include "config.h"

#include <stdio.h>

#include "helloworld-1.0/phrase.h"

int main (int argc, char ** argv)
{
    (void)argc;
    (void)argv;

    helloworld::Phrase phrase ("Hello World!");

    fprintf (stdout, "%s\n", phrase.getText().c_str());

    return 0;
}

tests/phrase_test.cpp:

#include <cppunit/BriefTestProgressListener.h>
#include <cppunit/CompilerOutputter.h>
#include <cppunit/extensions/HelperMacros.h>
#include <cppunit/extensions/TestFactoryRegistry.h>
#include <cppunit/TestResultCollector.h>
#include <cppunit/TestRunner.h>
#include <cppunit/TestResult.h>
#include <cppunit/ui/text/TestRunner.h>

#include <helloworld-1.0/phrase.h>

class PhraseTest : public CPPUNIT_NS::TestFixture
{
    CPPUNIT_TEST_SUITE (PhraseTest);
    CPPUNIT_TEST (testPhrase);
    CPPUNIT_TEST_SUITE_END ();

public:
    void setUp ();
    void tearDown ();

    void testPhrase ();
};

CPPUNIT_TEST_SUITE_REGISTRATION (PhraseTest);

void PhraseTest::setUp ()
{
}

void PhraseTest::tearDown ()
{
}

void PhraseTest::testPhrase ()
{
    std::string str = "Hello World!";
    helloworld::Phrase phrase (str);
    CPPUNIT_ASSERT_EQUAL (str, phrase.getText ());
    CPPUNIT_ASSERT ("" != phrase.getText ());

    std::string empty = "";
    phrase.setText (empty);
    CPPUNIT_ASSERT_EQUAL (std::string (empty), phrase.getText ());
    CPPUNIT_ASSERT (str != phrase.getText ());
}

int main (int argc, char ** argv)
{
    (void)argc;
    (void)argv;

    // informs test-listener about testresults
    CPPUNIT_NS::TestResult testresult;

    // register listener for collecting the test-results
    CPPUNIT_NS::TestResultCollector collectedresults;
    testresult.addListener (&collectedresults);

    // register listener for per-test progress output
    CPPUNIT_NS::BriefTestProgressListener progress;
    testresult.addListener (&progress);

    // insert test-suite at test-runner by registry
    CPPUNIT_NS::TestRunner testrunner;
    testrunner.addTest
        (CPPUNIT_NS::TestFactoryRegistry::getRegistry ().makeTest ());
    testrunner.run (testresult);

    // output results in compiler-format, with a custom error format that works
    // with a top-level Makefile.am and include.am's
    CPPUNIT_NS::CompilerOutputter compileroutputter (&collectedresults,
                                                     std::cerr,
                                                     "%p:%l:");
    compileroutputter.write ();

    // return 0 if tests were successful
    return collectedresults.wasSuccessful () ? 0 : 1;
}

Ok that’s it. You can now run autorun.sh, configure, and make and you “should” end up with a working shared library, pkg-config file, and executable. To test it all out do:

    $ ./config/autorun.sh
    $ ./configure --prefix=/tmp/helloworld-test
    $ make
    $ make check
    $ make install

Congrats, you now have a skeletal pandora-build based system. You can go in and add/remove shared libraries, headers, execs, tests, … as needed mostly by copy-n-pasting the above code and stuff you can find in other projects (both libdrizzle and drizzle are good, albeit it complicated, examples.) Feel free to ask questions if you have them and be sure to subscribe as I plan to make this a series of posts on the subject. Some of the planned topics include what needs to be in my repository (.gitignores,) integrating valgrind and other code checking/debugging utilities, logging, distribution/tarring things up, and maybe even building debian and redhat packages.

Rails/ActiveRecord::Base acts_as_obfuscated – hide your object ids

September 27th, 2009

I’ve always liked the idea of obfuscating the ids of my ActiveRecord::Base objects when using them in url. There’s lots of ways to go about this and through past projects I’ve used various mechanisms to accomplish it. Sometimes it’s nice and others it’s required. You might want to hide the number of users/objects/whatevers you have in you system or make it harder for malicious users to gain access to resources by guessing urls.

Well regardless of the reason(s) I’ve created a simple plugin/mixin, acts_as_obfuscated, that does exactly this with a single line of code. An example is in order:

class User < ActiveRecord::Base
  acts_as_obfuscated

  ...
end

and that's it.

$ ./script/console
Loading development environment (Rails 2.3.4)
>> u = User.create(:name => 'Bob')
=> #
>> u.id
=> 4
>> u.eid
=> "diBGnp"
>> User.find(u.id)
=> #
>> User.find(u.eid)
=> #
>>

The piece that's not shown above is an implementation of to_param.

def to_param
    self.eid
end

The effect of this is that anywhere you provider a user object in an 'id => user' param you get the self.eid rather than the default to_param of self.id. So a url that would look like http://mysite.com/users/4 would become http://mysite.com/users/diBGnp.

=link_to(user.name, :controller => 'users', :action => 'show', :id => user)

<a href="http://mysite.com/users/diBGnp">Bob</a>

acts_as_obfuscated doesn't get int the way of custom to_param functions so long as the first portion is the acts_as_obfuscated to_param function:

class User < ActiveRecord::Base
  acts_as_obfuscated

  def to_param
    CGI.escape("#{super.to_param}-#{self.name}").gsub(/\./, '_')
  end
end

$ ./script/console
Loading development environment (Rails 2.3.4)
u= U>> u= User.last
=> #
>> u.to_param
=> "diBGnp-Bob"
>> User.find(u.to_param)
=> #

That will allow you to have seo/ad placement friendly urls without exposing your internal object identifiers.

Anyway, check it out, use it, let me know what you think. The code can be found on github: http://github.com/ross/acts_as_obfuscated.

It can be installed as a plugin by running the command:

./script/plugin install git://github.com/ross/acts_as_obfuscated.git

If you want to see it in action check out: ClBrow - A Visual way to Shop Craigslist, which is a bit slower than I'd like to do the load on my dreamhost db, but hopefully I'll get around to fixing that soon...

-rm

5 Minute Custom Wordpress Theme

April 18th, 2009

Creating a custom wordpress theme is really easy especially if you’re starting with a template (a site design you want to use.) This is often the case when you’re trying to add a blog to an existing site or convert an existing website you’re happy with to use wordpress for blogging/content management. I won’t go in to details about how to install wordpress, there’s already a great guide for that. I’ll just outline the steps involved in creating a custom them to get wordpress to look & feel the way you want it to.

Laying the Foundation – Creating the Theme Directory

We’ll start out creating a directory to house our theme files (there’s only going to be 2 of them.) To do that we’ll log on to our server and execute the following:

$ cd /whereever/wordpress/is/installed/wp-content/themes
$ mkdir custom

If you don’t have shell access to you server use whatever mechanism you’ve uploaded/edited the sites files with in the past.

Step One – HTML – index.php

There’s only two files required to create a template for wordpress, index.php and style.css. We’ll start with index.php. To create an initial version of this file we’ll pick up where we left off a minute ago and do the following. If you’re familiar with another editor, feel free to use it, choices include vi, emacs, … but pico is one of the simplest to use (ctrl-O to save, ctrl-X to exit, more commands are listed across the bottom.)

$ cd custom
$ pico index.php

That will start up pico editing a file named index.php, the main template file for you new custom theme. We’ll start with a simple html page you can copy-n-paste in to this file.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" <?php language_attributes(); ?>>
  <head>
    <meta http-equiv="Content-Type" content="<?php bloginfo('html_type'); ?>;
      charset=<?php bloginfo('charset'); ?>" />
    <title>
      <?php wp_title('«', true, 'right'); ?> <?php bloginfo('name'); ?>
    </title>
    <link rel="stylesheet" href="<?php bloginfo('stylesheet_url'); ?>"
      type="text/css" media="screen" />
    <link rel="alternate" type="application/rss+xml"
      title="<?php bloginfo('name'); ?> RSS Feed"
      href="<?php bloginfo('rss2_url'); ?>" />
    <link rel="alternate" type="application/atom+xml"
      title="<?php bloginfo('name'); ?> Atom Feed"
      href="<?php bloginfo('atom_url'); ?>" />
    <link rel="pingback" href="<?php bloginfo('pingback_url'); ?>" />
  </head>
  <body>
    <div id='header'>
      Insert Your Header HTML here
    </div>
    <div id='content' class='span-16 prepend-1'>
        <?php if (have_posts()) : ?>
          <?php while (have_posts()) : the_post(); ?>
            <div <?php post_class() ?> id="post-<?php the_ID(); ?>">
              <h2><a href="<?php the_permalink() ?>" rel="bookmark"
                title="Permanent Link to <?php the_title_attribute();
                ?>"><?php the_title(); ?></a></h2>
              <small>
                <?php the_time('F jS, Y') ?>
                <!-- by <?php the_author() ?> -->
              </small>
              <div class="entry">
                <?php the_content('Read the rest of this entry »'); ?>
              </div>
              <p class="postmetadata"><?php the_tags('Tags: ', ', ', '<br />');
                ?> Posted in <?php the_category(', ') ?> |
                <?php edit_post_link('Edit', '', ' | '); ?>
                <?php comments_popup_link('No Comments »',
                                          '1 Comment »',
                                          '% Comments »'); ?>
              </p>
            </div>
          <?php endwhile; ?>
            <div class="navigation">
              <div class="alignleft"><?php
                next_posts_link('« Older Entries') ?></div>
              <div class="alignright"><?php
                previous_posts_link('Newer Entries »') ?></div>
            </div>
        <?php else : ?>
          <h2 class="center">Not Found</h2>
          <p class="center">Sorry, but you are looking for something that isn't
            here.</p>
          <?php get_search_form(); ?>
        <?php endif; ?>
      </div>
      <div id='sidebar' class='span-6 last'>
        <?php get_sidebar(); ?>
      </div>
    <div id='footer'>
      Insert Your Footer HTML here
    </div>
  </body>
</html>

Don’t worry if that looks like a mass of gibberish, there’s only a small portion of it that you’ll have to worry about, the two sections in red. In them you will place your header and footer HTML, whatever logos and/or text you’d like to see at the top of the page. If you’d like to have a navagation bar across the page you can create a second div following the header dive and put links to the various sections of you site there. The footer is a good place to put a copyright notice, links to email you or any other information you’d like to have appear at the bottom of all of your pages.

If you’re working with an existing template you want to insert wordpress into, you’ll take the section in blue and place it in the content section of your template. You may have to mess around with it a bit to get exactly what you’re looking for, but keep at it it shouldn’t take too long.

Step Two – CSS – style.css

We’re half the way to a new custom Wordpress theme. The next thing we’ll need to do is create style.css.

$ pico style.css

At this point if you want to save the file you can go to the admin section for you blog and click on the ‘Appearance’ link and you should see your new ‘custom’ theme. Clicking on it should pop up a preview of what your blog will look like using this theme. It probably won’t look like much yet, but it’s a nice clean workspace in which you’ll be able to mold things to your liking. If you don’t have visitors to your blog yet, or don’t mind them seeing the work in progress you may go head and apply your new theme. If you’re not ready for that you’ll need to continue to use the preview feature to view your work.

So one of the biggest problems with this theme is the sidebar is way down at the bottom below all of the content. We’ll need to add some css to address this issue, luckily there’s not much to it, at least to move the sidebar up. You’ll just need to add the following to style.css and refresh.

#header
{
}

#content
{
  float: left;
  width: 600px;
}

#sidebar
{
  float: left;
  width: 200px;
}

#footer
{
  clear: both;
}

The css above makes both #content and #sidebar float left and then limits #content’s width to 600 pixels and the sidebar to 200. So the blog will be 800 pixles wide. The only other thing going on is that we’ve asked the footer to clear both, which essentially means that it should go below any floating divs before it. This is obviously pretty rudimentary and doesn’t do much for the ascetic appeal of our blog, but everything “works” from here it’s just fiddling with css (which is way beyond the scope of this post.) Take a look at the HTML generated by this theme using view source and you should be able to track down the id’s and class’s you need to address to shape things up. Web developer Tool-bar can be really helpful for this work, check it out.

Conclusions

So we’ve created a simple, although still ugly, wordpress theme from scratch. It uses lots of defaults that can be customized to your liking, but it’s a good start. If you have any questions feel free to hit me up at rwmcfa1 <at> neces.com. Don’t have the time and/or desire to mess with custom wordpress instalation/development get in touch.

Integrating FreeBSD, ZFS, and Periodic; snapshots and scrubs

April 15th, 2009

ZFS on FreeBSD is powerful, especially when coupled with periodic taking hourly, daily, weekly, monthly, … snapshots. In the following post I’ll provide the scripts & config necessary to customize and walk you step-by-step through setting up zfs snapshots and scrubs with periodic on FreeBSD.

Periodic’s main advantage over the more traditional and obvious method of running a script from a cron job is integration with the notification emails and standard configuration mechanism. That may not sound like much, but that means you a year down the road (or someone else that comes to the system) only has to look in the obvious place to figure out what’s going on or make changes.

This is going to be a long post, but there’s a decent amount of code & config to walk through. The files being discussed have been tar’d up and the latest version of them can be downloaded from here.

Configuration – the stuff you might actually want to muck with

We’ll start with the configuration (/etc/periodic.conf) as it’s the most relevant portion or at least the most likely to be edited. Out of the box FreeBSD supports daily, weekly, and monthly periodic tasks, we’re going to be adding an hourly so that we can do hourly snapshots. The first section of config sets up who output from the hourly script should go to, whether it should be sent if everything succeeded, if something failed, or if something is mis-configured. Hourly emails seem a bit much so we’ve disabled them when everything goes well. We do want messages about errors and I’ve just left badconfig to the same value as all of the other time-frames.

# Hourly options
hourly_output="root"					# user or /file
hourly_show_success="NO"				# scripts returning 0
hourly_show_info="YES"					# scripts returning 1
hourly_show_badconfig="NO"				# scripts returning 2

The next section configures hourly snapshots. In this case we’re enabling them for the pool tank and keeping the 6 most recent around. There are defaults, that we’ll see later, for both pools and keep so the only required value here is enable. To specify more than one pool add them space seperated to the config string, e.g. “tank boat plane”

# 000.zfs-snapshot
hourly_zfs_snapshot_enable="YES"
hourly_zfs_snapshot_pools="tank"
hourly_zfs_snapshot_keep=6

The daily section is almost identical, but we instead keep the last 7 days. We’re also enabling a daily zfs status script that is in the default setup, but disabled.

# Daily options

# 000.zfs-snapshot
daily_zfs_snapshot_enable="YES"
daily_zfs_snapshot_pools="tank"
daily_zfs_snapshot_keep=7

# 404.status-zfs
daily_status_zfs_enable="YES"

Weekly and Monthly have the same configuration options, we’re keeping the last 5 weeks, and last 2 months below.

# Weekly options

# 000.zfs-snapshot
weekly_zfs_snapshot_enable="YES"
weekly_zfs_snapshot_pools="tank"
weekly_zfs_snapshot_keep=5

# Monthly options

# 000.zfs-snapshot
monthly_zfs_snapshot_enable="YES"
monthly_zfs_snapshot_pools="tank"
monthly_zfs_snapshot_keep=2

A final section configures the monthly scrubbing. Similarlly to the snapshot config sections there’s an enable line and pools line. Here we’ve enabled the monthly scrub on the tank pool.

# 998.zfs-scrub
monthly_zfs_scrub_enable="YES"
monthly_zfs_scrub_pools="tank"

periodic hourly – adding hourly script support to periodic

The next thing we need to do is add hourly support to periodic. While that might sound complicated it’s actually very straightforward. We start by creating a directory to house the hourly files.

# mkdir /etc/periodic/hourly

And then add the following line to /etc/crontab just before the line for ‘periodic hourly’

1	*	*	*	*	root	periodic hourly

That’s it you now have a place to put scripts that will be run every hour on the :01.

hourly/daily/weekly/monthly scripts – adding hourly script support to periodic

Now we’ll get to the scripts that make all of this configuration do something. It’s unlikely that you’ll ever have to much with any of these, but in case your curious I’ll go ahead and walk though them. We’ll start with the hourly snapshot script (/etc/periodic/hourly/000.zfs-snapshot.)

 1  #!/bin/sh
 2
 3  # If there is a global system configuration file, suck it in.
 4  #
 5  if [ -r /etc/defaults/periodic.conf ]
 6  then
 7      . /etc/defaults/periodic.conf
 8      source_periodic_confs
 9  fi
10
11  pools=$hourly_zfs_snapshot_pools
12  if [ -z "$pools" ]; then
13      pools='tank'
14  fi
15
16  keep=$hourly_zfs_snapshot_keep
17  if [ -z "$keep" ]; then
18      keep=6
19  fi
20
21  case "$hourly_zfs_snapshot_enable" in
22      [Yy][Ee][Ss])
23          . /etc/periodic/zfs-snapshot
24          do_snapshots "$pools" $keep 'hourly'
25          ;;
26      *)
27          ;;
28  esac

Lines 3-9 is boilerplate periodic script stuff. 11-19 look for the values we configured earlier and use defaults if they’re not specified. 21, 22, 25, and 26 are case shell scripting case statement stuff that’s borrowed from one of the other periodic scripts, mainly just makes sure that hourly_zfs_snapshot_enable is set to YES, ignoring case. Line 32 pulls in (think #include) some common zfs snapshotting code that we’ll get to next and finally line 24 calls the snapshotting function for the configured pools, keep count, and the type of hourly. The daily, weekly, and monthly scripts are identical with hourly replaced with the appropriate value throughout.

zfs-snapshot – the workhorse

There’s too much here to walk though in detail so I’ll let you read through the code. I’ve tried to do a decent job of in-line commenting. If you have questions or want clarification feel free to ask…

#!/bin/sh

# checks to see if there's a scrub in progress
scrub_in_progress()
{
  pool=$1

  if zpool status $pool | grep "scrub in progress" > /dev/null; then
    return 0
  else
    return 1
  fi
}

# take the appropriately named snapshot
create_snapshot()
{
    pool=$1

    case "$type" in
        hourly)
        now=`date +"$type-%Y-%m-%d-%H"`
        ;;
        daily)
        now=`date +"$type-%Y-%m-%d"`
        ;;
        weekly)
        now=`date +"$type-%Y-%U"`
        ;;
        monthly)
        now=`date +"$type-%Y-%m"`
        ;;
        *)
        echo "unknown snapshot type: $type"
        exit 1
    esac

    # create the now snapshot
    snapshot="$pool@$now"
    # look for a snapshot with this name
    if zfs list -H -o name | sort | grep "$snapshot$" > /dev/null; then
        echo "	snapshot, $snapshot, already exists"
    else
        echo "	taking snapshot, $snapshot"
        zfs snapshot -r $snapshot
    fi
}

# delete the named snapshot
delete_snapshot()
{
    snapshot=$1
    echo "	destroying old snapshot, $snapshot"
    zfs destroy -r $snapshot
}

# take a type snapshot of pool, keeping keep old ones
do_pool()
{
    pool=$1
    keep=$2
    type=$3

    # create the regex matching the type of snapshots we're currently working
    # on
    case "$type" in
        hourly)
        # hourly-2009-01-01-00
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]$"
        ;;
        daily)
        # daily-2009-01-01
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]$"
        ;;
        weekly)
        # weekly-2009-01
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]"
        ;;
        monthly)
        # monthly-2009-01
        regex="$pool@$type-[0-9][0-9][0-9][0-9]-[0-9][0-9]"
        ;;
        *)
        echo "unknown snapshot type: $type"
        exit 1
    esac

    create_snapshot $pool $type

    # get a list of all of the snapshots of this type sorted alpha, which
    # effectively is increasing date/time
    # (using sort as zfs's sort seems to have bugs)
    snapshots=`zfs list -H -o name | sort | grep $regex`
    # count them
    count=`echo $snapshots | wc -w`
    if [ $count -ge 0 ]; then
        # how many items should we delete
        delete=`expr $count - $keep`
        count=0
        # walk through the snapshots, deleting them until we've trimmed deleted
        for snapshot in $snapshots; do
            if [ $count -ge $delete ]; then
                break
            fi
            delete_snapshot $snapshot
            count=`expr $count + 1`
        done
    fi
}

# take snapshots of type, for pools, keeping keep old ones,
do_snapshots()
{
    pools=$1
    keep=$2
    type=$3

    echo ""
    echo "Doing zfs $type snapshots:"
    for pool in $pools; do
        if scrub_in_progress $pool; then
          echo "	skipping snapshot of $pool, scrub in progress"
        else
          do_pool $pool $keep $type
        fi
    done
}

ZFS snapshots reset/restart zfs scrub on FreeBSD 7.0

April 12th, 2009

I ran in to an interesting problem that took me a little bit of digging/thinking to figure out this morning. When you do a zpool scrub on FreeBSD and then do zfs snapshot the scrub restarts from the beginning, (at least as of this writing.) I couldn’t find any references to this and FreeBSD, but did find mentions of zfs scrub resetting with OSX. The same thread claimed the issue was addressed in OpenSolairs versions past that which had been ported/integrated in to OSX, probably the same for FreeBSD. I started a scrub last night before I went to bed and woke up this morning and it had less % done than it did the last time I looked, that’s not good. I checked in on it every few minutes for a while and noticed that it was completing pretty fast, but then kicked back up after a while. It didn’t immediately occur to me that it was at the top of the hour when my snapshot script had just run, but I got there a few minutes later.

At any rate, once I knew what was going on it made perfect sense and was pretty easily addressed. As part of addressing it I’ve generalized my snapshot setup and converted it to be run by periodic. A post with that is coming as soon as I live with it for a bit and make sure the kinks are worked out. It has actually turned out really cool, hourly, daily, weekly, monthly snapshots, monthly scrubs (during which snapshots are suspected and it’s all configurable…

ZFS snapshots — poor man’s backup — solution to the ‘rm *’ whoops problem

April 4th, 2009

Daily/Hourly Snapshots Script

First off snapshots != backups, but they’re still really useful. A solution to the whoops I didn’t mean to do that problem. Similar to the benefits of having a delayed slave with mysql, if you accidentally do something to mess up your world/data you can go back in time a little bit and “undo” it. ZFS gives you great tools for doing file system snapshots and makes recovering from problems possible. I use the following simple script I coded up on my newly built FreeBSD 7.0 full ZFS system.

#!/bin/sh

ZFS=zfs
POOL=tank

# i run this in a cron tab with output redirected to a
# log file, this gives me a ran at time
date

# delete the last hourly snapshot, the old one that
# should go away, in this case 2 hours ago
LASTHOUR=`date -v-2H +"%Y-%m-%d-%H"`
echo "deleting hourly snapshot $LASTHOUR of $POOL"
$ZFS destroy -r $POOL@$LASTHOUR
# create the new snapshot, this hour
HOURLY=`date +"%Y-%m-%d-%H"`
echo "taking hourly snapshot $HOURLY of $POOL"
$ZFS snapshot -r $POOL@$HOURLY

# at 12:00 utc, ~ 4am local time, i run this server in utc
if [ `date +"%H"` = '12' ]; then
  # same as LASTHOUR, but 1 week ago
  LASTWEEK=`date -v-1w +"%Y-%m-%d"`
  echo "deleting daily snapshot $LASTWEEK of $POOL"
  $ZFS destroy -r $POOL@$LASTWEEK
  # todays
  DAILY=`date +"%Y-%m-%d"`
  echo "taking daily snapshot $DAILY of $POOL"
  $ZFS snapshot -r $POOL@$DAILY
fi

It takes hourly snapshots, keeping around the last 2, and daily snapshots keeping around the last 7. That’s good enough to suit my purposes, but the script could easily be appended to take more frequent snapshots or to do longer term snapshotting: weekly, monthly, …

Disk Usage

One awesome thing is that if you have data that is added to, never taken away, snapshots won’t take up any extra space. If you make changes then both copies of the data will need to be stored and thus double the space will be required. Deleting, won’t free up the space so long as the snapshot lives. With my use-case I see about 3% overhead for snapshots on my frequently used pieces and nearly 0% everywhere else.

Viewing Current Usage

Useful commands for taking a peek at snapshots and space usage:

# zfs list -t filesystem
# zfs list

The first will show only filesystems, no snapshots etc. It’s similar in purpose to df. The second will show all filesystem and snapshot usage. This one tells you how much space each is taking up. If you have evenly spaced snapshots over time it can give you an idea how much churn you’re seeing.

An example of the output from zfs list for one of my filesystems and it’s snapshots

tank/media                54.7G  2.80T  47.9G  /media/media
tank/media@2009-03-29         0      -  46.3G  -
tank/media@2009-03-30         0      -  46.3G  -
tank/media@2009-03-31       43K      -  47.5G  -
tank/media@2009-04-01     1.20G      -  49.3G  -
tank/media@2009-04-02      350M      -  46.5G  -
tank/media@2009-04-03       52K      -  47.4G  -
tank/media@2009-04-04       48K      -  47.1G  -
tank/media@2009-04-05-04      0      -  47.9G  -
tank/media@2009-04-05-05      0      -  47.9G  -

Reading through the above the overall usage is at 54.7G. Nothing has changed in the past 2 hours. Tiny little bits of data have changed in the last two days and the two days before that saw decent size chunks changing at 350M and 1.2G. This is a pretty common pattern for this filesystem for me. The snapshots give me a place to go when I accidentally delete a file or make an unintended change.

Getting At Your Snapshot Data

Oh, one more thing… You can get at the snapshots using the .zfs directory. So if the tank/media filesystem was mounted at /media/ you’d find yesterday’s snapshot at /media/.zfs/snapshots/2009-04-03.

FreeBSD ZFS Root, running a full system in ZFS

March 23rd, 2009

I’ll start with a little bit of back story, if you’re just interested in the tech skip ahead…

FreeBSD and Me

FreeBSD and I have a long history together, in fact just about as long as I’ve been playing with computers. It was ~1996, I had just graduated from high school and managed to find the account sign up page for the University of Kentucky. I signed up for all of the accounts it would give me having no idea what they were since I didn’t start school for another 4 months or so. One of those accounts turned out to be email, pop.uky.edu. The server provided telnet access so I logged on and was greeted with a motd including FreeBSD. Up until that point I’d only been exposed to Windows. I knew that UNIX systems existed, but didn’t really have access to one. I could tell this was a UNIX system and the word Free piqued my interest. It turned out that it was a dual-P2 with a few hundred megabytes of RAM. I was pretty amazed, a pretty modest system hosting 40k users email accounts. A few minutes later I was downloading several (probably 8-12) floppy images and on my way to having a UNIX of my own. I’ve haven’t liked windows since.

I recently purchased a new system primarily intended as a network storage server. It’s a fairly modest, or at least cheap, system in all respects except hard drive space, I have 3 identical Segate drives weighing in at 1.5 TB each. I knew I wanted to give ZFS a try so I started looking at Open Solaris. I never really liked the Solaris env and quickly found Nexenta which seemed to address that complaint. Problem turned out to be that Nexenta is only avaliable in 32-bit flavors, wtf is up with that, and therefore didn’t cope with 4G of RAM or 1.5TB HD (I can’t remember which.) In reading around about ZFS I had run across something talking about ZFS support in FreeBSD. In reality I should of started there, but I wanted to mess around with DTrace as well, little did I know that also has been ported to FreeBSD. So an hour or so later I had a working FreeBSD system, fully running on ZFS.
</backstory>

Requirements

  • A system with a spare 3G or more in its own partition (though you’ll want way more for it to be interesting.)
  • FreeBSD bootonly iso, burnt on to a CDROM (or another installation method)
  • A couple hours to play around with it all

I initally worked off of the process described here, what follows is pretty similar and changes only where I’ve optimized out a step or two or just done things according to my situation, ymmv. As always please read my disclaimer thoroughly.

My Setup

  • 3 1.5TB hard drives in fully dedicated disk mode (the only part that’s really relevant)
  • New Egg shared wish list for the full details.
  • Old PCI 3-com Ethernet card as the NIC on the motherboard didn’t seem to get along with FreeBSD.

FreeBSD Install

I won’t walk through every detail of my process, but some of the big ones include

  1. Use entire disk for all 3 disks, installing bootloader on all 3
  2. Create a 1G partition on each disk, and allocate the rest to a 2nd partition, the important thing here, since I’m using raidz, is that the involved partitions are identical in size.
    • ad4s1a: 1G UFS, / (during install anyway)
    • ad4s1d: rest of drive, initially set to mount at /zfs0, but immeidately removed the mount point (takes a long time to format the file system and we don’t need it to be)
    • ad6s1a: 1G UFS, /r2, will eventually be used as a redundant/backup root
    • ad6s1d: same as ad4s1d
    • ad8s1b: 1G swap
    • ad8s1d: same as ad4s1d
  3. Otherwise install whatever you like, I always start with a minimal install and add things as I need/want them.

Post Install – The Real Stuff

After removing the disk and rebooting select single user mode:

mount -w /

Create the raid pool, allocating whatever partitions you have to use, my command looked like:

zpool create tank raidz /dev/ad3s1d /dev/ad6s1d /dev/ad8s1d

We don’t want to mount the pool as a whole, so unset it’s mountpoint

zfs set mountpoint=none tank

Now we’ll create our root filesystem in the tank pool. We’re specifying a temporary mount point that we can use for now, later we’ll disable mounting for this filesystem as it will be mounted before booting from it.

zfs create -o mountpoint=/tank tank/root

This is just an example of how you would create other filesystems inside of the tank pool. Its safe to create home and go ahead and mount it in its final location so long as you didn’t create users during the intial install process. If you did then you need to do something similar to the root copy we’ll be doing in a second to copy the data from the old partition to the new.

zfs create -o mountpoint=/home tank/home

Check out what you’ve created with

df -h
zfs list

Now enable ZFS on boot

echo 'zfs_enable="YES"' >> /etc/rc.conf

And then copy all of the current, non-ZFS, root data over to the ZFS root partition.

find -x / | cpio -pmd /tank 

No remove the boot directory we just copied over

rm -rf /tank/boot

And then create a point at which the UFS boot directory will be mounted so that we can update it later if need be.

mkdir /tank/bootdir
cd /tank
ln -s bootdir/boot boot

Now we need to tell the UFS bootloader to enable ZFS and to boot from our root ZFS filesystem. We also need to do some ZFS tuning based on information found in the wiki at ZFSTuningGuide, check it out for more info on what’s being set there. The specific values depend mainly on the amount of RAM you have and what (else) you’ll be using the box for.

echo '
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
#vfs.zfs.arc_max="100M"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"
' >> /boot/loader.conf

Edit the ZFS root’s fstab, /tank/etc/fstab, so that the UFS bootdir is mounted where we pointed the symlink up above, the dev portion of this may vary if you’ve

/dev/ad4s1a /bootdir	ufs	rw	1	1

And finally change the root filesystem’s mountpoint so that ZFS won’t try to mount it automatically.

cd /
zfs set mountpoint=legacy tank/root

Thanks pretty much it, just reboot and check it out.

The last step is to mirror the UFS boot parition on to a 2nd drive in case the first dies, to do this

find -x /bootdir | cpio -pmd /r2

Any time you want to add a new filesystem, they can be very handy for bookkeeping and concise snapshotting, you just do something like the following

zfs create -o mountpoint=/media/music tank/music