| View previous topic :: View next topic |
| Author |
Message |
Simon
Guest
|
 Posted: Sun Nov 16, 2008 4:50 am Post subject: [gentoo-user] Binary package cruncher? |
 |
|
Hey there!
I've purchased a linode some time ago because my hardware is very (*very*)
old and it was taking very long to do updates. Now, what I've done is I've
setup my linode to be exactly the same as my host, portage-wise. And on the
linode, I have installed the sum of all packages that are installed on all my
PCs at home... this way, when doing a `emerge -uDN world` on the linode, all
packages are updated for all my machines (no fancy scripts to compile each
machine's pkg independantly, etc).
So, this is a binary package maker (or cruncher as it does it really well!).
However, one detail that is interesting is that to make sure all compilations
work fine, I decided to really install everything on the host, yes, including
stuff that requires a X server and so on (which I never use on the host).
This is fine with me except it's taking a lot of space to have all that
around... so, since my host is really fast, i thought it might not be a problem
for me to build dependencies in a sandbox, build the program, keep only the
binpkg and remove everything including the dependencies that "had to be
installed" for proper compilation.
Ideally, I would see my file /var/lib/portage/world to contain strictly the
software used on the host (ie. apache, mysql, etc...) and another file like
/var/lib/portage/binworld which would contain all that is in world plus all the
other packages that are in the worlds of my other PCs. Packages in binworld
would be compiled, as well as their dependancies. However, if a binpkg is
available for a dependancy, then this one is used instead. And this way, all
the "ghost" dependancies and "ghost" packages will all be compiled exactly as it
is now, with some overhead for decompressing the deps binpkg for each pkg
compilation.
This is not super efficient, but the way I've thought it, should be simple to
do... a simple gentoo hack so to speak. However, I'm wondering if anybody has
suggestion for better ways to do this and if you could give me pointers to such
projects. Also keep in mind that I really really want to update ALL my PCs with
a single `emerge -uDN world` on the host, then copy new pkgs (using rsync or
other) to the PCs and do an `emerge -k -uDN world` on them. Nothing more. (So
unless your suggestion is simpler than my current (fully installed) setup,
please tell me!)
Thanks in advance!
Simon |
|
| |
|
 Back to top |
Alan McKinnon
Guest
|
| Posted: Sun Nov 16, 2008 4:10 pm Post subject: Re: [gentoo-user] Binary package cruncher? |
|
|
On Saturday 15 November 2008 19:40:50 Simon wrote:
| Quote: | This is not super efficient, but the way I've thought it, should be
simple to do... a simple gentoo hack so to speak. However, I'm wondering
if anybody has suggestion for better ways to do this and if you could give
me pointers to such projects. Also keep in mind that I really really want
to update ALL my PCs with a single `emerge -uDN world` on the host, then
copy new pkgs (using rsync or other) to the PCs and do an `emerge -k -uDN
world` on them. Nothing more. (So unless your suggestion is simpler than
my current (fully installed) setup, please tell me!)
|
I would build all that stuff in a chroot. The logic is that the your buildhost
system is not quite the same thing as the machine hosting the buildhost.
One thing you cannot get away from is that to build say X for your slow
hardware, it has to be done on a machine that has all X's build dependencies
fully installed and working. You might not want that on your production
server. Some fancy tricks with bind-mounts into the chroot would let you
share common stuff. Or, you could simply buy a bit more storage if you are
running out. It's cheap enough and if you've gone to this much trouble
already, some more storage would be minor
--
alan dot mckinnon at gmail dot com |
|
| |
|
| Back to top |
Peter Humphrey
Guest
|
| Posted: Sun Nov 16, 2008 5:20 pm Post subject: Re: [gentoo-user] Binary package cruncher? |
|
|
On Sunday 16 November 2008 10:05:53 Alan McKinnon wrote:
| Quote: | On Saturday 15 November 2008 19:40:50 Simon wrote:
This is not super efficient, but the way I've thought it, should be
simple to do... a simple gentoo hack so to speak. However, I'm
wondering if anybody has suggestion for better ways to do this and if
you could give me pointers to such projects. Also keep in mind that I
really really want to update ALL my PCs with a single `emerge -uDN
world` on the host, then copy new pkgs (using rsync or other) to the
PCs and do an `emerge -k -uDN world` on them. Nothing more. (So
unless your suggestion is simpler than my current (fully installed)
setup, please tell me!)
I would build all that stuff in a chroot. The logic is that the your
buildhost system is not quite the same thing as the machine hosting the
buildhost.
One thing you cannot get away from is that to build say X for your slow
hardware, it has to be done on a machine that has all X's build
dependencies fully installed and working. You might not want that on your
production server. Some fancy tricks with bind-mounts into the chroot
would let you share common stuff. Or, you could simply buy a bit more
storage if you are running out. It's cheap enough and if you've gone to
this much trouble already, some more storage would be minor
|
Another possibility is to use distcc. I haven't used it recently, but an
earlier version seemed ok.
--
Rgds
Peter |
|
| |
|
| Back to top |
Simon
Guest
|
| Posted: Mon Nov 17, 2008 2:10 am Post subject: Re: [gentoo-user] Binary package cruncher? |
|
|
| Quote: | I would build all that stuff in a chroot. The logic is that the your buildhost
system is not quite the same thing as the machine hosting the buildhost.
|
That's the big trick I was thinking about...
1) Make a proper development environment under a chroot, place the "binworld" on
top of world.
2) emerge -k -uDN world
(this will install, in the chroot env all dependencies using previously
built binpkgs, if those dependencies don't exist, they will be built and binpkg
will be made for everything that is built)
3) un-chroot properly (unbind directories, etc)
4) delete the chroot directory containing installed dependencies, not relevant
to the production environment
This would build the packages and install nothing on the host. But I'm not sure
if chroot is really proper for this... maybe I would have to use a unionfs to
make sure that nothing gets overwritten in /lib or gets installed in the real
/lib for example...
After everything is built, i would just redo the `emerge -k -uDN world` on the
host in the real production environment to update it.
| Quote: | One thing you cannot get away from is that to build say X for your slow
hardware, it has to be done on a machine that has all X's build dependencies
fully installed and working. You might not want that on your production
server. Some fancy tricks with bind-mounts into the chroot would let you
share common stuff. Or, you could simply buy a bit more storage if you are
running out. It's cheap enough and if you've gone to this much trouble
already, some more storage would be minor
|
You're right and this is why I had installed X on my host, with all its deps,
even though none of it was used. But using the chroot (and/or unionfs trick)
I'd be able to build it with its deps without really installing it on the prod
environment.
However, diskspace on a VPS is very costly. For example, one extra GB is
2$/m... and with the cheapest VPS i have only 12GB now. I'm sure my full
production environment would not move much higher than 4GB (without counting
/usr/portage) while it probably is filling my disk now...
So, basically, if gentoo is a meta-distribution, my host is "my" distribution
and serves to build the packages that are then available to my PCs...
Thanks,
Simon |
|
| |
|
| Back to top |
Simon
Guest
|
| Posted: Mon Nov 17, 2008 2:10 am Post subject: Re: [gentoo-user] Binary package cruncher? |
|
|
| Quote: | Another possibility is to use distcc. I haven't used it recently, but an
earlier version seemed ok.
|
Distcc is used to speed compilation time. But the point is that even if I set
distcc hosts to not be on the localhost (ie, just compile remotely and only use
localhost to manage the whole thing)... it still takes way too long. Plus,
many packages don't compile with distcc, will compile localhost only. Plus,
many parts of the emerge are not distributed, like ./configure, etc...
I mean, installing a binpkg is already slow on my PCs, imagine the compilation
(yea, i am very patient). That's why I purchased the VPS to compile and make
binpkg... One day I might purchase a second VPS and set distcc between the two
(a binpkg farm?)... but until then distcc is really near useless in my setup.
Only exception to what I just said is with the kernel, which I always compile
with distcc.
Thanks,
Simon |
|
| |
|
| Back to top |
|
