Getting started with pkgsrc in Owl

Getting started with pkgsrc in Owl

The NetBSD Packages Collection (pkgsrc) is a framework for building third-party software on NetBSD and other UNIX-like systems.

Official manual says so and much more. I strongly encourage to read them, keep them always open, consult them in first place. Text version is located in doc/ directory of pkgsrc tree.

Pros and cons of pkgsrc

Strictly subjective strong and weak sides of pkgsrc.

Pros

OS-agnostic nature. It's not FreeBSD project, it's not Gentoo or FreeBSD project. pkgsrc is usable in various linux distros, Haiku, DragonFlyBSD and even Minix. pkgsrc installation is also separated from main system and can be removed or archived by one command without impact to base system.
pkgsrc has list of known vulnerabilities, maintained by NetBSD Security Officer and pkgsrc team (pkgsrc manual)
automatic builds: every package-specific options can be easily set and unset in global configuration file and packages can be build with all dependencies by single command. User-specific packages groups can be created with so-called metapackages.
good documentation and large, friendly community.

Cons

pkgsrc often uses libraries and programs from base-system (for example zlib, curses) but sometimes not (typical example: perl). This leads to redunancy (two perls or something like that)
enlisting vulnerability doesn't mean fixing them in pkgsrc, but bumping versions and rebuilding package is simply dead
pkgsrc policy doesn't enforce backporting fixes (like Debian does), this is up to maintainer - often fixing vulnerability means sync with upstream version with all consequences (like config file changes)

Preparing environment

I strongly recommend creating separate environment especially for pkgsrc builds. Now pkgsrc supports builds to separate DESTDIR (like rpm does) and building from unprivileged user, but dedicated environment still saves us from installing not needed in production, build-time dependencies and is, in my opinion, more secure.

Creating OpenVZ container (most natural in Owl) is beyond scope of this document. Some hilights:

standard Owl template is good enough
we need working network connections - pkgsrc automagically fetch sources during build phase
we need /usr/pkg/bin and /usr/pkg/sbin in our PATH
pkgsrc prefers umask 022, not 077 as in Owl default
if we are paranoid and don't use root account to build (pkgsrc has hashes for sources, but we can't trust unknown sources and build scripts, right?) we also needs unprivileged account. When we build as unprivileged user we must remember about privileges on certain directories.
whole pkgsrc tree needs 880 megabytes of space (but we needs more space: for builds, sources and packages)

Getting and upgrading pkgsrc tree

Initial fetching and updating pkgsrc tree is covered by pkgsrc manual. Personally, I prefer working with development branch (aka -current). Typically pkgsrc tree is located in /usr/pkgsrc but it's only convention: /home/builder/pkgsrc is good enough.

In this example I assume default location for pkgsrc source tree, in /usr/pkgsrc.

Fetching and unpacking:

$ lftp ftp://ftp.NetBSD.org/pub/pkgsrc/current/
cd ok, cwd=/pub/pkgsrc/current                                           
lftp ftp.NetBSD.org:/pub/pkgsrc/current> mget pkgsrc.tar.bz2*

...

$ sha1sum -c pkgsrc.tar.bz2.SHA1
pkgsrc.tar.bz2: OK
$ tar -C /usr -xjf pkgsrc.tar.bz2

We may also want update recently unpacked snapshot:

$ cd /usr/pkgsrc
$ cvs -q update -dP

Bootstrapping pkgsrc

At start we need basic tools used by pkgsrc.

$ cd /usr/pkgsrc/bootstrap

As root:

$  ./bootstrap --workdir /tmp/pkgsrc --pkgdbdir=/usr/pkg/var/db --binary-kit /home/builder/pkgsrc-bootstrap.tar

…or as unprivileged user:

$  ./bootstrap --workdir /tmp/pkgsrc --pkgdbdir=/usr/pkg/var/db --unprivileged --prefix /usr/pkg --binary-kit /home/builder/pkgsrc-bootstrap.tar

Explaining:

–workdir sets working directory for bootstrapping process (only), pkgsrc can be served even from read-only medium, so I use location in /tmp, but it's optional
–pkgdbdir it's bigger violation from standard behaviour: typically pkgsrc locates binaries in /usr/pkg and pkgsrc meta-data in /var/db directory. Having both in /usr/pkg it's convinient - whole installation can be moved, archived on erased by one command. I frequently forgot about deleting both /usr/pkg and /var/db and finally decide to keep meta-data under /usr/pkg
–unprivileged required only if we build as ordinary user
–prefix only required in unprivileged builds, without that pkg destination is located in $HOME. Of course /usr/pkg must be created before and owned by unprivileged account
–binary-kit produces tarball with basic tools, that allow us easily bootstrapping pkgsrc in production environment, as mentioned above

Configuring pkgsrc

Whole configuration of pkgsrc relies on single /usr/pkg/etc/mk.conf file. Personally, in build environment, I prefer keeping them in /etc/mk.conf (as in NetBSD) and make symbolic link from /usr/pkg/etc/mk.conf to /etc/mk.conf. It's protects our precious config from accidental removing with whole /usr/pkg. In production environment it's doesn't matter, default mk.conf, included in previously created bootstrap tarball, is good enough and doesn't needs our attention.

Below my mk.conf. For normal uses default is enough, but my personal may serve as example:

.ifdef BSD_PKG_MK       # begin pkgsrc settings

# no special options for gcc-3 in owl
CFLAGS+=        -O2 -pipe -march=i686
CXXFLAGS+=      -O2 -pipe -march=i686

# this is only for FreeBSD
#BUILDLINK_PREFIX.ncurses=/usr

# standard directiories
WRKOBJDIR=              /var/pkgsrc/work
DISTDIR=                /var/pkgsrc/distfiles
PACKAGES=               /var/pkgsrc/packages
RCD_SCRIPTS_DIR=        /etc/init.d
LOCALBASE=              /usr/pkg
PKG_DBDIR=              /usr/pkg/var/db
PKG_TOOLS_BIN=          /usr/pkg/sbin
VARBASE=                /var
PKGMANDIR=              man
#
#
PKG_DEFAULT_OPTIONS+=   inet6
PKG_VERBOSE=            no
USE_XPKGWEDGE=          yes
PKG_RCD_SCRIPTS=        yes
FAILOVER_FETCH=         yes               # insist on the correct checksum
PKG_DEVELOPER?=         yes               # not needed for ordinary users
SKIP_LICENSE_CHECK=     yes
X11_TYPE=               modular
#UNPRIVILEGED=           yes               # only with unprivileged account
#
USE_DESTDIR=            yes               # redundant in current -current
DEPENDS_TARGET=         package-install
UPDATE_TARGET=          package-install

# application options
PKG_UID.nginx=  1982             
PKG_GID.nginx=  1982             
PKG_UID.slapd=  1983             
PKG_GID.ldap=   1983             
                                
PHP_VERSION_DEFAULT=53          
PGSQL_VERSION_DEFAULT=84        
MYSQL_CHARSET=utf8              
PYTHON_VERSION_DEFAULT=python27 
VIM_EXTRA_OPTS=--enable-langmap 

#
PKG_OPTIONS.nginx=              ssl pcre inet6
PKG_OPTIONS.php=                inet6 -suhosin
PKG_OPTIONS.screen=             ncurses
PKG_OPTIONS.mysql5=             -embedded-server -ndb-cluster -pstack -ssl
PKG_OPTIONS.mc=                 slang -charset
PKG_OPTIONS.ssmtp=              -ssl
PKG_OPTIONS.postfix=            pcre tls
PKG_OPTIONS.mutt=               ssl mutt-hcache ncursesw mutt-hcache
.endif                  # end pkgsrc settings

As we see mk.conf utilizes makefile syntax (NetBSD make, to be strict), it's included during build process, and we are able to utilize more complex constructions, like:

USE_SSP?=               yes                                                                           
USE_FORT?=              yes                                                                           
MKPIE?=                 no                                                                            
MKRELRO?=               yes                

.if ${.CURDIR:M*/wip/php-fpm} || \                           
    ${.CURDIR:M*/www/nginx} || \                             
    ${.CURDIR:M*/lang/php53}                                 
MKPIE=yes                                                    
.endif                                                       
                                                             
# 
CFLAGS+=        -O2 -pipe -march=i686                        
CXXFLAGS+=      -O2 -pipe -march=i686                        
                                                             
.if ${USE_SSP} != "no"                                       
CFLAGS+=        -fstack-protector -fstack-protector-all      
CFLAGS+=        -Wstack-protector --param ssp-buffer-size=1  
.endif                                                       
                                                             
.if ${USE_FORT} != "no"                                      
CFLAGS+=        -D_FORTIFY_SOURCE=2                          
.endif                                                       
                                                             
.if ${MKPIE} != "no"                                         
CFLAGS+=        -fPIC -DPIC                                  
LDFLAGS+=       -Wl,-pie -shared-libgcc                      
.endif                                                       
                                                             
.if ${MKRELRO} != "no"                                       
LDFLAGS+=       -Wl,-z,relro -Wl,-z,now                      
.endif

Following section needs additional explanation:

WRKOBJDIR=              /var/pkgsrc/work
DISTDIR=                /var/pkgsrc/distfiles
PACKAGES=               /var/pkgsrc/packages

Typically all data resides under pkgsrc directory: sources in distfiles/ subdir, created packages in packages/ and work directiories in individual package's dirs. I prefer keeping them separated from “clean” pkgsrc tree and place under /var/pkgsrc. It's convinent for me, but this is only suggestion.

Building package

Before build starts we must update information about vulnerable packages:

$ pkg_admin -vv fetch-pkg-vulnerabilities
Fetching http://ftp.NetBSD.org/pub/NetBSD/packages/vulns/pkg-vulnerabilities.gz

Then we can build something simple. Under linux we use 'bmake' command for that:

$ cd /usr/pkgsrc/mail/mutt-devel 

/usr/pkgsrc/mail/mutt-devel $ bmake show-options

Any of the following general options may be selected:
        debug    Enable debugging facilities in the package.
        gpgme    Enable gpg support, by using the gpgme library.
        idn      Internationalized Domain Names (IDN) support.
        mutt-compressed-mbox    Enable compressed mailbox support in mutt.
        mutt-hcache      Enable header caching in mutt.
        mutt-sidebar     Enable the mutt mailbox sidebar patch.
        mutt-smtp        Enable SMTP support in mutt.
        sasl     Enable SASL support.
        smime    Enable S/MIME support.
        ssl      Enable SSL support.
Exactly one of the following display options is required:
        curses   Enable curses support.
        ncurses  Enable ncurses support.
        ncursesw         Enable wide character ncurses support.
        slang    Enable S-Lang support.

These options are enabled by default:
        curses smime ssl

These options are currently enabled:
        mutt-hcache ncursesw smime ssl

You can select which build options to use by setting PKG_DEFAULT_OPTIONS
or PKG_OPTIONS.mutt.

/usr/pkgsrc/mail/mutt-devel $ bmake package

=> Bootstrap dependency digest>=20010302: found digest-20080510
=> Bootstrap dependency tnftp-[0-9]*: found tnftp-20070806
WARNING: [license.mk] Every package should define a LICENSE.
===> Checking for vulnerabilities in mutt-1.5.21nb1
=> Checksum SHA1 OK for mutt-1.5.21.tar.gz
=> Checksum RMD160 OK for mutt-1.5.21.tar.gz
===> Installing dependencies for mutt-1.5.21nb1

[ whole mess there, and finally... ]

===> Building binary package for mutt-1.5.21nb1
=> Creating binary package /home/builder/pkgsrc/packages/All/mutt-1.5.21nb1.tgz
===> Leaving ``package'' after barrier for mutt-1.5.21nb1
$

Warning: if you use bmake package command, package isn't installed yet. It's can be achieved by using bmake package-install instead of package target or later, by pkg_add /home/builder/pkgsrc/packages/All/mutt-1.5.21nb1.tgz.

Bootstrapping in production

First, read using pkgsrc section (or, better, whole manual ;). Then create production environment, set PATH as mentioned in previous sections and unpack previously created tarball:

# tar -C / -xf pkgsrc-bootstrap.tar
# pkg_admin -vv fetch-pkg-vulnerabilities

Next step depends: you may copy selected packages to temporary directory, mount them with mount –bind or export them via http or ftp. Then, you can set PKG_PATH environment variable to something like

ftp://[user]:[password]@[host]/pkgsrc/packages/All

and simply add package:

# pkg_add mutt
pkg_add: Warning: package `mutt-1.5.21nb1' was built for a platform:
pkg_add: Linux/i386 2.6.18 (pkg) vs. Linux/i386 2.6.18-238.5.1.el5.028stab085.1 (this host)
pkg_add: Warning: package `perl-5.12.2nb2' was built for a platform:
pkg_add: Linux/i386 2.6.18 (pkg) vs. Linux/i386 2.6.18-238.5.1.el5.028stab085.1 (this host)
pkg_add: Warning: package `ncursesw-5.9' was built for a platform:
pkg_add: Linux/i386 2.6.18 (pkg) vs. Linux/i386 2.6.18-238.5.1.el5.028stab085.1 (this host)
pkg_add: Warning: package `ncurses-5.9' was built for a platform:
pkg_add: Linux/i386 2.6.18 (pkg) vs. Linux/i386 2.6.18-238.5.1.el5.028stab085.1 (this host)
pkg_add: Warning: package `db4-4.8.30' was built for a platform:
pkg_add: Linux/i386 2.6.18 (pkg) vs. Linux/i386 2.6.18-238.5.1.el5.028stab085.1 (this host)
mutt-1.5.21nb1: copying /usr/pkg/share/examples/mutt/Muttrc to /usr/pkg/etc/Muttrc
mutt-1.5.21nb1: copying /usr/pkg/share/examples/mutt/mime.types to /usr/pkg/etc/mime.types
mutt-1.5.21nb1: setting permissions on /usr/pkg/bin/mutt_dotlock (o=builder, g=mail, m=2551)

As you can see all dependencies are resolved automagically. Complains about platform difference (Linux/i386 2.6.18 (pkg) vs. Linux/i386 2.6.18-238.5.1.el5.028stab085.1) should be ignored, this is known, already fixed in HEAD, non-problematic issue.

…and remember of periodically calling pkg_admin audit-pkg!

Table of Contents