Helio/packages
From FBSD_tips
A (not-so) Novel Package Management Scheme (note, please excuse any errors, I'm rebuilding this from almost a year old memory of a prototype I developed)
Contents |
[edit] Structure
This system is designed to NFS export a single directory, which serves as the 'program repository' for several heterogenous machines. It's intended to support multiple parallel versions, concurrent builds, and multiple active 'profiles'. Multiple ABIs and processor architectures can be built for each program, and handled transparently. Note: this is just an example that is almost entirely workable, there are a few semantics that would need to be addressed.
Each 'package' is a subdirectory under /repo/programs, named after an sha256 hash of the main program, and all it's built dependencies, for example:
/repo/programs/22394e543da05b80cc58e36b417e3eff5f56658582a2ce2e3368dc72103c952d/
is the root for nano, inside it is the following structure:
/FreeBSD-6.2/ /FreeBSD-6.2/i386/ /FreeBSD-6.2/i386/ /FreeBSD-6.2/i386/db/pkg/ /FreeBSD-6.2/i386/db/ports/ /FreeBSD-6.2/i386/lib/ /FreeBSD-6.2/i386/bin/ /FreeBSD-6.2/i386/share/ /FreeBSD-6.2/amd64/ /FreeBSD-6.2/amd64/db/pkg/ /FreeBSD-6.2/amd64/db/ports/ /FreeBSD-6.2/amd64/lib/ /FreeBSD-6.2/amd64/bin/ /FreeBSD-6.2/amd64/share/ /distfiles
distfiles is where all the program (and it's dependencies) distfiles are stored, while db/pkg and db/ports contain the list of packages inside the program, and it's port options, respectively.
There is a directory named /repo/map containing a single file, mapfile, of this format:
nano-2.0.6=22394e543da05b80cc58e36b417e3eff5f56658582a2ce2e3368dc72103c952d
the 'nano-2.0.6' portion is /not/ a unique key, multiple hash's may exist for a version; these need to be interrogated by the system by examining their db/pkg/ directory for the differences.
Also, there is a /repo/profile directory, containing multiple named files, each containing a newline terminated list of sha256 hashes for that profile. Directories are allowed, the name is merely concantenated to /repo/profile/ when it's specified.
[edit] File consolidation
One of the questions about this format, is consolidation of shared files. Namely, there was none. Until Now.
If during install, we extract to a staging area (let's say, /repo/stage/HASH); then, we run through each file; we sha256 every file we use. Then, we check if a /repo/files/HASH exists; if it doesn't, we create a link to the file in that directory, named after our hash.
Then, in both cases, we link the /repo/files/HASH to the logical name in the /repo/programs/ directory, and unlink the file in the staging area.
Multiple identical files will be consolidated, and files can be safely deleted from /repo/files/ when their link count reaches '1'. However, it would be pointless to do it all the time, just occasionally.
The biggest downside, is that ports by default do not build libs and binaries so that they are identical when the net effect is the same.
[edit] Using ports as just the build infrastructure
The ports system is surprisingly capable for a build infrastructure for third-party programs on FreeBSD systems. However, it needs some work to properly detach it from the metadata.
cd /usr/ports/editors/nano
export REPODIR=${HOME}/repo/$(cat * | sha256)/$(uname)-$(uname -r | cut -d - -f 1)/$(uname -p)
export DISTDIR=${REPODIR}/distfiles
export PREFIX=${REPODIR}
export PORT_DBDIR=${REPODIR}/db/ports
export PKG_DBDIR=${REPODIR}/db/pkg
mkdir -p ${PKG_DBDIR}
mkdir -p ${PORT_DBDIR}
mkdir -p ${WRKDIRPREFIX}
mkdir -p ${DISTDIR}
make install
[edit] The Glue
Add the NFS exported directory to your fstab, under /repo.
Also it would be good to add a unioned cache directory on top of the repository, like so:
mkdir /cache/1 mount -t nullfs -o union,ro,noatime /cache/1 /repo
Let's add a second cache directory so we can update the cache, and mark it for use on the next boot, without stepping on the current cache.
mkdir /cache/2
In this way you can add some logic to the on boot profile reading code, to locally cache copies of binaries in the repo that you use, for when the NFS server is down.
Let's go through the shell script that implements all of this.
First, let's grab a profile name from kenv (if it exists), this will let you set a profile in /boot/loader.conf to override what's in the machine's rc.conf:
boot_profile="$(kenv -q boot_profile)"
Now, let's set PROFILE to either what's set in loader.conf, what's set in rc.conf, or nowhere.
if [ "${boot_profile}" = "" ] ; then
if [ "${PROFILE}" = "" ] ; then
PROFILE=default
fi
else
PROFILE=${boot_profile}
fi
now, let's use PROFILE to merge what we have in the repo, into our namespace:
ABI="$(uname)-$(uname -r | cut -d - -f 1)"
ARCH="$(uname -p)"
for i in $(cat /repo/profile/${PROFILE})
do
mount -o union,ro,noatime -t nullfs /repo/${1}/${ABI}/${ARCH}/bin /usr/local/bin
done
note we did no error checking, it's up to you to do error checking at this point.
[edit] Conclusion
I hope this is enough to demonstrate that package management need not always be concerned with tracking dependencies, rather improving organization. I also hope that it gets the point across that collision detection in package management has never been handled as cleanly as possible.
For an example of a similar design, take a look at: [1]. Though there are some differences in their profile implementation, it is close to my original idea. However, as far as I can tell, there is no attempt at file consolidation.
