Created at:

Modified at:

pkgsrc bulkbuild

This file documents my efforts to make pkgsrc bulkbuild work.

Two very important resources are the bulkbuild chapter_ of the pkgsrc guide_ and the bulkbuild HOWTO_ in the pkgsrc "inner" documentation. Also, take a look at the pbulk wiki HOWTO_.

bulkbuild chapter of the pkgsrc guide

pkgsrc guide

bulkbuild HOWTO

pbulk wiki HOWTO

In this guide modern pbulk is made instead of classic bulkbuild, as shown in the bulkbuild chapter of the pkgsrc guide, although the terminology "bulkbuild" is used.

Setting up the environment

I setup a machine only for bulkbuilds, because I didn't want to setup a chroot environment in one of the work machines (the bulkbuild chapter of the pkgsrc guide recommends to either set a special machine or a chroot'ed environment, because bulkbuilds usually mess things up, invalidating the machine for anything that is not building packages).

In this machine, I downloaded pkgsrc and untared it in the default place: /usr/pkgsrc: 

    $ cd /tmp
    $ ftp ftp://ftp.netbsd.org/pub/pkgsrc/pkgsrc-2009Q2/pkgsrc.tar.gz
    $ su -
    # cd /tmp
    # tar zxf pkgsrc.tar.gz -C /usr

The guide says, yet, that it is necessary to bootstrap pkgsrc, even if NetBSD is being used. So:: 

    # cd /usr/pkgsrc/bootstrap
    # ./bootstrap --prefix /usr/pbulk --varbase /usr/pbulk/var

*Important*: If you are trying to run pbulk in a non-NetBSD environment, you need to make a bootstrap kit as told in bulkbuild HOWTO of the pkgsrc guide. So, after the previous bootstrap command, execute the following commands:: 

        # ./cleanup
        # ./bootstrap --mk-fragment /tmp/mk.conf \
            --gzip-binary-kit bootstrap_kit.tar.gz

If you cannot find the tarball, where is it? See if it is in /.

Then, install the pbulk tools:: 

    # cd /usr/pkgsrc/pkgtools/pbulk
    # /usr/pbulk/bin/bmake install

For more information on bootstrapping pkgsrc, check the pkgsrc-bootstrap page:

pkgsrc-bootstrap

Configuring

All configuration is made by a single file, located in ${prefix}/etc/pbulk.conf, in our case, /usr/pbulk/etc/pbulk.conf.

Important changes to this file are:

Example for a limited_list file: 

audio/bladeenc
audio/cdparanoia
chat/irssi
devel/scmgit
editors/vim
lang/sbcl
mail/mutt
misc/screen
security/sudo
www/elinks
www/firefox3

"bulkbuilding"

The environment and configuration are set up.

*Note*: Before executing the following steps, I had problems making bulkbuild finding bmake in the right directory, although I set the make variable correctly in pbulk.conf. To make it work, I needed to:: # export PATH=/usr/pkg/bin:/usr/pkg/sbin:${PATH}

So, let's build the packages:: 

    # cd /usr/pbulk/bin
    # ./bulkbuild

It will scan for the packages and start to build them. It will deliver the built packages where the PACKAGES variable (in /etc/mk.conf) points to, defaults to /usr/pkgsrc/packages.

Most errors can arrive in this step. Check the Troubleshooting section.

Since the bulkbuild is complete, if you change your limited_list file and run ./builbuild script again, it will build all the packages in the limited_list, including those that are already built. To build only the packages you added, follow the following steps:

1. Editor the limited_list file normally.

2. Run the scan fase of the pbulk:: 

    # cd /usr/pbulk/libexec/pbulk
    # ./scan

3. Run the build fase:: 

    # ./build

Troubleshooting

Bulkbuild reports complain: pkg_install is too old

The version of pkg_admin you have installed in the base system is probably too old. If you bootstrapped pkgsrc, it compiled a newer version of pkg_admin for you. You have to tell pkgsrc that you want to use the bootstrapped version:: 

    # export PKG_TOOLS_BIN=/usr/pbulk/sbin

More information about bulk builds environment variables in pkgsrc/doc/mk/bulk/environment.

warning: mail_queue_enter: create file maildrop/89559.19302: No such ...

Postfix is probably not working. Try /etc/rc.d/postfix forcestart. Before doing that, check whether or not you need it (read the bulkbuild HOWTO again). If not, make rsync=: and mail=: in pbulk.conf.

Build packages fail with depends.log having digest package

Does the variable PACKAGES in mk.conf points to the same place where the variable packages in pbulk.conf points to?

They should point to the same place.

Build some packages fail with indirect-prefailed

(2010-01-06)

Check report.html. Does it tell that you that pkgsrc needs X11? Did you forgot to install the X11 sets in your NetBSD installation?

Error "tnftp (or whatever) already installed"

(2009-07-29)

You may get an error like:: 

    ===> Installing for tnftp-20070806
    ERROR: tnftp-20070806 is already installed - perhaps an older version?
    ERROR: If so, you may use either of:
    ERROR:     - "pkg_delete tnftp-20070806" and "/usr/pbulk/bin/bmake reinstall"
    ERROR:       to upgrade properly
    ERROR:     - "/usr/pbulk/bin/bmake update" to rebuild the package and all
    ERROR:       of its dependencies
    ERROR:     - "/usr/pbulk/bin/bmake replace" to replace only the package without
    ERROR:       re-linking dependencies, risking various problems.
    *** Error code 1

And /usr/pbulk is your bootstrapped pkgsrc, you are probably wondering why it is calling /usr/pbulk/bin/bmake and not /usr/pkg/bin/bmake. Check if you set make variable correctly in pbulk.conf according to your architecture.

Some other interesting references

Besides links on the text, there are other references worth to have a look:

Chapter about bulkbuids on pkgsrc guide

Bulkbuild with pkgsrc page on DragonFlyBSD documentation

Error "Unresolvable dependencies found" with pbulk

(2011-04-18)

If you get an error like:: 

    Unresolvable dependencies found, exiting:
    databases/py-sqlite3

databases/py-sqlite3 exists. What is the problem, then? On the scan/resolve phase of a bulkbuild (pbulk), it is very likely to be a version mismatch problem.

In this case I had to add this line, to avoid version 2.4 of Python being considered:: 

    PYTHON_VERSIONS_ACCEPTED=27 26 25

Joerg Sonnenberger here helped me with this problem:

Re: Error "Unresolvable dependencies found" with pbulk

(2021-12-17)

Yet another example:: 

    Unresolvable dependencies found, exiting:
    textproc/py-pygments

It means that a package depends on textproc/py-pygments but it could be any other dependency. In summary it means that the pbulk system could not find this dependency. You can find it more in the files generated by pbulk, that are in the meta directory. As we'll see, the problem is more complicated in this case, because Python packages can be built for different versions of Python. Let's see a snippet of the meta/presolve file regarding this dependency: 

    PKGNAME=py27-docutils-0.16nb1
    PKG_LOCATION=textproc/py-docutils
    ALL_DEPENDS=  digest>=20010302:../../pkgtools/digest  cwrappers>=20150314:../../pkgtools/cwrappers   py27-expat>=0:../../textproc/py-expat py27-pygments>=0:../../textproc/py-pygments py27-roman>=1.4:../../math/py-roman py27-setuptools-[0-9]*:../../devel/py-setuptools44  python27>=2.7:../../lang/python27  python27>=2.7.1nb2:../../lang/python27

So, we are trying to build py27-docutils that depends on py27-pygments, but there is no package py27-pygments (i.e., py-pygments can be compiled for Python 3.6, Python 3.7, etc., but not for Python 2.7).

In this case, it may be that a package is adding a dependency that doesn't exist in pkgsrc (a pkgsrc bug), or some inconsistency in your local tree.

I'm using a rather old tree (2021Q3) and it seems already fixed in current tree, but I decided to make my own change and just added the following line to textproc/py-docutils: 

    PYTHON_VERSIONS_INCOMPATIBLE=	27

And it fixed the problem.

Error Cyclic dependency for package: cwrappers-... digest-... cwrappers-...

(2018-11-09)

When trying a new bulkbuild I got this error:: 

    Cyclic dependency for package:
    cwrappers-20180325
    digest-20160304
    cwrappers-20180325
    *** Error code 1

The reason is that I was wrongly setting variable DISTFILES in /etc/mk.conf, whereas the correct one is DISTDIR.