Created at:
Modified at:
pkgsrc bulkbuild
This file documents my efforts to make pkgsrc bulkbuild work.
Two very important resources are the bulkbuild chapter
_ of the pkgsrc
guide
_ and the bulkbuild HOWTO
_ in the pkgsrc "inner" documentation. Also,
take a look at the pbulk wiki HOWTO
_.
bulkbuild chapter of the pkgsrc guide
In this guide modern pbulk is made instead of classic bulkbuild, as shown in the bulkbuild chapter of the pkgsrc guide, although the terminology "bulkbuild" is used.
Setting up the environment
I setup a machine only for bulkbuilds, because I didn't want to setup a chroot environment in one of the work machines (the bulkbuild chapter of the pkgsrc guide recommends to either set a special machine or a chroot'ed environment, because bulkbuilds usually mess things up, invalidating the machine for anything that is not building packages).
In this machine, I downloaded pkgsrc and untared it in the default place:
/usr/pkgsrc
:
$ cd /tmp
$ ftp ftp://ftp.netbsd.org/pub/pkgsrc/pkgsrc-2009Q2/pkgsrc.tar.gz
$ su -
# cd /tmp
# tar zxf pkgsrc.tar.gz -C /usr
The guide says, yet, that it is necessary to bootstrap pkgsrc, even if NetBSD is being used. So::
# cd /usr/pkgsrc/bootstrap
# ./bootstrap --prefix /usr/pbulk --varbase /usr/pbulk/var
*Important*: If you are trying to run pbulk in a non-NetBSD environment, you need to make a bootstrap kit as told in bulkbuild HOWTO of the pkgsrc guide. So, after the previous bootstrap command, execute the following commands::
# ./cleanup
# ./bootstrap --mk-fragment /tmp/mk.conf \
--gzip-binary-kit bootstrap_kit.tar.gz
If you cannot find the tarball, where is it? See if it is in /
.
Then, install the pbulk tools::
# cd /usr/pkgsrc/pkgtools/pbulk
# /usr/pbulk/bin/bmake install
For more information on bootstrapping pkgsrc, check the pkgsrc-bootstrap page:
Configuring
All configuration is made by a single file, located in
${prefix}/etc/pbulk.conf
, in our case, /usr/pbulk/etc/pbulk.conf
.
Important changes to this file are:
- Set
master_mode=no
if you don't want to use multiple machines for building the packages. - Set
bootstrapkit=
if you are using NetBSD (according to the bulkbuild HOWTO). - Set
limited_list=/limited_list
to the path to the file that you want to specify what packages you want to build. For example:limited_list=/home/user/pbulk-limited_list
. And, in the file, put the new-line separated list of packages you want to build (see an example just after the end of this list). - pkgsrc now has
user-destdir
enabled by default and you should tell inpbulk.conf
what is the unprivileged user you want it to use to build pacakges in theunprivileged_user
variable. Create this user manually (if it doesn't exist yet) and make sure destination directory of packages and pkgsrc both have their owner set to this user. - If you are using NetBSD, set
make=/usr/bin/make
. - If you don't want to publish your packets automatically, set
mail=:
andrsync=:
. I forgot to do that and it produced a lot of non-clear problems.
Example for a limited_list
file:
audio/bladeenc
audio/cdparanoia
chat/irssi
devel/scmgit
editors/vim
lang/sbcl
mail/mutt
misc/screen
security/sudo
www/elinks
www/firefox3
"bulkbuilding"
The environment and configuration are set up.
*Note*: Before executing the following steps, I had problems making
bulkbuild finding bmake
in the right directory, although I set the
make variable correctly in pbulk.conf. To make it work, I needed to:: #
export PATH=/usr/pkg/bin:/usr/pkg/sbin:${PATH}
So, let's build the packages::
# cd /usr/pbulk/bin
# ./bulkbuild
It will scan for the packages and start to build them. It will deliver the
built packages where the PACKAGES
variable (in /etc/mk.conf
) points
to, defaults to /usr/pkgsrc/packages
.
Most errors can arrive in this step. Check the Troubleshooting section.
Since the bulkbuild is complete, if you change your limited_list file and run
./builbuild
script again, it will build all the packages in the
limited_list, including those that are already built. To build only the
packages you added, follow the following steps:
1. Editor the limited_list
file normally.
2. Run the scan fase of the pbulk::
# cd /usr/pbulk/libexec/pbulk
# ./scan
3. Run the build fase::
# ./build
Troubleshooting
Bulkbuild reports complain: pkg_install is too old
The version of pkg_admin you have installed in the base system is probably too old. If you bootstrapped pkgsrc, it compiled a newer version of pkg_admin for you. You have to tell pkgsrc that you want to use the bootstrapped version::
# export PKG_TOOLS_BIN=/usr/pbulk/sbin
More information about bulk builds environment variables in
pkgsrc/doc/mk/bulk/environment
.
warning: mail_queue_enter: create file maildrop/89559.19302: No such ...
Postfix is probably not working. Try /etc/rc.d/postfix forcestart
. Before
doing that, check whether or not you need it (read the bulkbuild HOWTO again).
If not, make rsync=:
and mail=:
in pbulk.conf
.
Build packages fail with
depends.log
having
digest
package
Does the variable PACKAGES
in mk.conf
points to the same place where
the variable packages
in pbulk.conf
points to?
They should point to the same place.
Build some packages fail with indirect-prefailed
(2010-01-06)
Check report.html
. Does it tell that you that pkgsrc needs X11? Did you
forgot to install the X11 sets in your NetBSD installation?
Error "tnftp (or whatever) already installed"
(2009-07-29)
You may get an error like::
===> Installing for tnftp-20070806
ERROR: tnftp-20070806 is already installed - perhaps an older version?
ERROR: If so, you may use either of:
ERROR: - "pkg_delete tnftp-20070806" and "/usr/pbulk/bin/bmake reinstall"
ERROR: to upgrade properly
ERROR: - "/usr/pbulk/bin/bmake update" to rebuild the package and all
ERROR: of its dependencies
ERROR: - "/usr/pbulk/bin/bmake replace" to replace only the package without
ERROR: re-linking dependencies, risking various problems.
*** Error code 1
And /usr/pbulk
is your bootstrapped pkgsrc, you are probably wondering why
it is calling /usr/pbulk/bin/bmake
and not /usr/pkg/bin/bmake
. Check
if you set make
variable correctly in pbulk.conf
according to your
architecture.
Some other interesting references
Besides links on the text, there are other references worth to have a look:
Chapter about bulkbuids on pkgsrc guide
Bulkbuild with pkgsrc page on DragonFlyBSD documentation
Error "Unresolvable dependencies found" with pbulk
(2011-04-18)
If you get an error like::
Unresolvable dependencies found, exiting:
databases/py-sqlite3
databases/py-sqlite3 exists. What is the problem, then? On the scan/resolve phase of a bulkbuild (pbulk), it is very likely to be a version mismatch problem.
In this case I had to add this line, to avoid version 2.4 of Python being considered::
PYTHON_VERSIONS_ACCEPTED=27 26 25
Joerg Sonnenberger here helped me with this problem:
Re: Error "Unresolvable dependencies found" with pbulk
(2021-12-17)
Yet another example::
Unresolvable dependencies found, exiting:
textproc/py-pygments
It means that a package depends on textproc/py-pygments
but it could be any
other dependency. In summary it means that the pbulk system could not find this
dependency. You can find it more in the files generated by pbulk, that are in
the meta
directory. As we'll see, the problem is more complicated in this
case, because Python packages can be built for different versions of Python.
Let's see a snippet of the meta/presolve
file regarding this dependency:
PKGNAME=py27-docutils-0.16nb1
PKG_LOCATION=textproc/py-docutils
ALL_DEPENDS= digest>=20010302:../../pkgtools/digest cwrappers>=20150314:../../pkgtools/cwrappers py27-expat>=0:../../textproc/py-expat py27-pygments>=0:../../textproc/py-pygments py27-roman>=1.4:../../math/py-roman py27-setuptools-[0-9]*:../../devel/py-setuptools44 python27>=2.7:../../lang/python27 python27>=2.7.1nb2:../../lang/python27
So, we are trying to build py27-docutils that depends on py27-pygments, but there is no package py27-pygments (i.e., py-pygments can be compiled for Python 3.6, Python 3.7, etc., but not for Python 2.7).
In this case, it may be that a package is adding a dependency that doesn't exist in pkgsrc (a pkgsrc bug), or some inconsistency in your local tree.
I'm using a rather old tree (2021Q3) and it seems already fixed in
current tree, but I decided to make my own change and just added the
following line to textproc/py-docutils
:
PYTHON_VERSIONS_INCOMPATIBLE= 27
And it fixed the problem.
Error Cyclic dependency for package: cwrappers-... digest-... cwrappers-...
(2018-11-09)
When trying a new bulkbuild I got this error::
Cyclic dependency for package:
cwrappers-20180325
digest-20160304
cwrappers-20180325
*** Error code 1
The reason is that I was wrongly setting variable DISTFILES
in
/etc/mk.conf
, whereas the correct one is DISTDIR
.