pkgsrc bulkbuild

This file documents my efforts to make pkgsrc bulkbuild work.

Two very important resources are the bulkbuild chapter of the pkgsrc guide and the bulkbuild HOWTO in the pkgsrc "inner" documentation. Also, take a look at the pbulk wiki HOWTO.

In this guide modern pbulk is made instead of classic bulkbuild, as shown in the bulkbuild chapter, although the terminology "bulkbuild" is used.

Setting up the environment

I setup a machine only for bulkbuilds, because I didn't want to setup a chroot environment in one of the work machines (the bulkbuild chapter recommends to either set a special machine or a chroot'ed environment, because bulkbuilds usually mess things up, invalidating the machine for anything that is not building packages).

In this machine, I downloaded pkgsrc and untared it in the default place: /usr/pkgsrc:

$ cd /tmp
$ ftp ftp://ftp.netbsd.org/pub/pkgsrc/pkgsrc-2009Q2/pkgsrc.tar.gz
$ su -
# cd /tmp
# tar zxf pkgsrc.tar.gz -C /usr

The guide says, yet, that it is necessary to bootstrap pkgsrc, even if NetBSD is being used. So:

# cd /usr/pkgsrc/bootstrap
# ./bootstrap --prefix /usr/pbulk --varbase /usr/pbulk/var

Important

If you are trying to run pbulk in a non-NetBSD environment, you need to make a bootstrap kit as told in bulkbuild HOWTO. So, after the previous bootstrap command, execute the following commands:

# ./cleanup
# ./bootstrap --mk-fragment /tmp/mk.conf \
    --gzip-binary-kit bootstrap_kit.tar.gz

If you cannot find the tarball, where is it? See if it is in /.

(For more information on bootstrapping pkgsrc, check the pkgsrc-bootstrap page).

Then, install the pbulk tools:

# cd /usr/pkgsrc/pkgtools/pbulk
# /usr/pbulk/bin/bmake install

Configuring

All configuration is made by a single file, located in ${prefix}/etc/pbulk.conf, in our case, /usr/pbulk/etc/pbulk.conf.

Important changes to this file are:

Set master_mode=no if you don't want to use multiple machines for building the packages.
Set bootstrapkit= if you are using NetBSD (according to the bulkbuild HOWTO).
Set limited_list=/limited_list to the path to the file that you want to specify what packages you want to build. For example:
```
limited_list=/home/user/pbulk-limited_list
```
And, in the file, put the new-line separated list of packages you want to build:
```
audio/bladeenc
audio/cdparanoia
chat/irssi
devel/scmgit
editors/vim
lang/sbcl
mail/mutt
misc/screen
security/sudo
www/elinks
www/firefox3
```
pkgsrc now has user-destdir enabled by default and you should tell in pbulk.conf what is the unprivileged user you want it to use to build pacakges in the unprivileged_user variable. Create this user manually (if it doesn't exist yet) and make sure destination directory of packages and pkgsrc both have their owner set to this user.
If you are using NetBSD, set make=/usr/bin/make.
If you don't want to publish your packets automatically, set mail=: and rsync=:. I forgot to do that and it produced a lot of non-clear problems.

"bulkbuilding"

The environment and configuration are set up.

Note

Before executing the following steps, I had problems making bulkbuild finding bmake in the right directory, although I set the make variable correctly in pbulk.conf. To make it work, I needed to:

# export PATH=/usr/pkg/bin:/usr/pkg/sbin:${PATH}

Let's build the packages:

# cd /usr/pbulk/bin
# ./bulkbuild

It will scan for the packages and start to build them. It will deliver the built packages where the PACKAGES variable (in /etc/mk.conf) points to, defaults to /usr/pkgsrc/packages.

Most errors can arrive in this step. Check the Troubleshooting section.

Since the bulkbuild is complete, if you change your limited_list file and run ./builbuild script again, it will build all the packages in the limited_list, including those that are already built. To build only the packages you added, follow the following steps:

Editor the limited_list file normally.
Run the scan fase of the pbulk:
```
# cd /usr/pbulk/libexec/pbulk
# ./scan
```
Run the build fase:
```
# ./build
```

Troubleshooting

Bulkbuild reports complain `pkg_install is too old`

The version of pkg_admin you have installed in the base system is probably too old. If you bootstrapped pkgsrc, it compiled a newer version of pkg_admin for you. You have to tell pkgsrc that you want to use the bootstrapped version:

# export PKG_TOOLS_BIN=/usr/pbulk/sbin

More information about bulk builds environment variables in pkgsrc/doc/mk/bulk/environment.

`warning: mail_queue_enter: create file maildrop/89559.19302: No such ...`

Postfix is probably not working. Try /etc/rc.d/postfix forcestart. Before doing that, check whether or not you need it (read bulkbuild HOWTO again). If not, make rsync=: and mail=: in pbulk.conf.

Build packages fail with `depends.log` having `digest` package

Does the variable PACKAGES in mk.conf points to the same place where the variable packages in pbulk.conf points to?

They should point to the same place.

Build some packages fail with `indirect-prefailed`

Check report.html. Does it tell that you that pkgsrc needs X11? Did you forgot to install the X11 sets in your NetBSD installation?

Error "tnftp (or whatever) already installed"

You may get an error like:

===> Installing for tnftp-20070806
ERROR: tnftp-20070806 is already installed - perhaps an older version?
ERROR: If so, you may use either of:
ERROR:     - "pkg_delete tnftp-20070806" and "/usr/pbulk/bin/bmake reinstall"
ERROR:       to upgrade properly
ERROR:     - "/usr/pbulk/bin/bmake update" to rebuild the package and all
ERROR:       of its dependencies
ERROR:     - "/usr/pbulk/bin/bmake replace" to replace only the package without
ERROR:       re-linking dependencies, risking various problems.
*** Error code 1

And /usr/pbulk is your bootstrapped pkgsrc, you are probably wondering why it is calling /usr/pbulk/bin/bmake and not /usr/pkg/bin/bmake. Check if you set make variable correctly in pbulk.conf according to your architecture and the bulkbuild HOWTO.

Some other interesting references

Besides links on the text, there are other references worth to have a look:

Chapter about bulkbuids on pkgsrc guide

Bulkbuild with pkgsrc page on DragonFlyBSD documentation

Error "Unresolvable dependencies found" with pbulk

If you get an error like:

Unresolvable dependencies found, exiting:
databases/py-sqlite3

databases/py-sqlite3 exists. What is the problem, then? On the scan/resolve phase of a bulkbuild (pbulk), it is very likely to be a version mismatch problem.

In this case I had to add this line, to avoid version 2.4 of Python being considered:

PYTHON_VERSIONS_ACCEPTED=27 26 25

Joerg Sonnenberger here helped me with this problem.

Error `Cyclic dependency for package: cwrappers-... digest-... cwrappers-...`

When trying a new bulkbuild I got this error:

Cyclic dependency for package:
cwrappers-20180325
digest-20160304
cwrappers-20180325
*** Error code 1

The reason is that I was wrongly setting variable DISTFILES in /etc/mk.conf, whereas the correct one is DISTDIR.