[packagers] Distributing the product

Dag Wieers dag at wieers.com
Mon Apr 18 17:18:00 CEST 2011

On Thu, 14 Apr 2011, David Hrbáč wrote:

> I'd like to sum up distribution aspects of RF. I can see three main items:
> 1. reorganise repository
> We want more mirrors, but our repository is quite messy. We begin with:
> [DIR] aurora/                   -   Aurora apt/yum tree
> [DIR] fedora/                   -   Fedora apt/yum tree
> [DIR] redhat/                   -   Red Hat apt/yum tree
> [DIR] source/                   -   Source RPM packages
> As far I remember, there's a plan to get rid of Fedora dir.
> Redhat contains of:
> [DIR]    6.2/    04-Mar-2008 20:53     -
> [DIR]    7.3/    04-Mar-2008 20:50     -
> [DIR]    8.0/    04-Mar-2008 20:50     -
> [DIR]    9/    04-Mar-2008 20:16     -
> [DIR]    el2.1/    04-Mar-2008 20:51     -
> [DIR]    el3/    04-Mar-2008 20:15     -
> [DIR]    el4/    04-Mar-2008 20:14     -
> [DIR]    el5/    04-Mar-2008 20:15     -
> [DIR]    el6/    10-Nov-2010 23:37     -
> The very same, plans are to get rid of 6.2,7.3,8.0,9, and el2.1. I'd
> like to have the repository clean and reorganised before we raise the
> call for the mirrors. The tree is over-combined. The final url is:
> http://apt.sw.be/redhat/el5/en/i386/rpmforge/. I guess we can remove en
> subfolder, we are not going to create any localised repositories. I'd
> like to see something like this:
> http://apt.sw.be/{el3,el4,el5,el6,sources}/{i386,x86_64,ppc}/{stable,testing,extras}/{RPMS,repodata,repoview}.
> I personally prefer:
> http://apt.sw.be/{el3,el4,el5,el6,sources}/{stable,testing,extras}/{SRPMS,i386,x86_64,debugi-386,i386,x86_64,debug-x86_64}/{repodata,repoview}

I agree, although I would keep el2.1 packages for the time being. The 
question is whether we want to keep th eold package somewhere in a 
publically accessible vault, or not. Maybe el2.1 belongs there.

Repoview in the past was very expensive to create, but if we drop APT 
metadata, and only offer 1 metadata per repository this would be possible. 
Still I would so love to have a one-pass tool that generates both metadata 
and interface.

On the other hand, having a dynamic web-interface that allows to 
browse/search the repository metadata would be prefered over a static 
repoview. For one, it does not have to be mirrored !!

> 2. sqlitise repository
> With respect to amount of packages we build, we really badly need sqlite
> enabled repositories. This is something we want for years, and we need
> it now.

Yes, but surya is RHEL4. Karanbir mentioned that he now can do sqlite on 
RHEL4, but it is currently still not possible. Either we move to another 
system, or we need Karanbir's solution.

> 3. distributing repository
> There are many options to go with repository distribution
> - rsync
> - mirrormanager
> - mirrorbrain
> For now I'd like to stick with rsync only. I'm not sure about our
> mirrors. http://apt.sw.be/redhat/el5/en/mirrors-rpmforge reads:
> http://apt.sw.be/redhat/el5/en/$ARCH/rpmforge
> http://ftp-stud.fht-esslingen.de/dag/redhat/el5/en/$ARCH/rpmforge
> http://fr2.rpmfind.net/linux/dag/redhat/el5/en/$ARCH/rpmforge
> while http://apt.sw.be/ reads:
>      Main Mirror
>      Bulgaria (BG): University of Sofia
>      Canada (CA): University Of Calgary
>      France (FR): Institute de Recherche et Coordination
> Acoustique/Musique, RPMfind.net
>      Germany (DE): Technische Universität Chemnitz, Universität Esslingen
>      Ireland (IE): Ireland's National Education & Research Network
>      United Kingdom (UK): UK Mirror Service
>      United States (US): Iowa State University

These mirrors are collected once by using Google. Instead of having a list 
of people that mirror it, we should be having a mailinglist and actively 
communicate with mirror admins. In fact, they should be part of this 
discussion before we redesign the repositories.

> thoughts:
> - Dag's blackbox :o), tree without repodata, signing packages, pushing
> the tree to master
> - master repository, not propagated to yum client, creating repodata,
> repoview (mirror.projectname.tld)
> - mirrors synced over rsync with master
> - create mirror maillist
> - monitor mirrors with mirmon (mirror-status.projectname.tld or
> mirror.projectname.tld/mirmon), I'm going to commit my mirmon spec file
> - start with static yum mirror list
> - clean up the mirror list, get in touch with those we have now
> - call for the mirrors
> - working on dynamic yum list based on mirmon status
> - with tens of mirrors, there no need for geoip,
> yum-fastestmirror-pluging should be enough for now
> - (metalinks,geoip,MirrorMnagaer, MirrorBrain)

There is one thing that I would like to bring into this discussion. I very 
much like the packages.sw.be interface. In fact I often use this at 
customers for downloading a specific package for a specific distribution.

I would like to keep this structure, in fact, it is the base structure 
used for creating everything else. The question begs if we want mirrors to 
also mirror this structure (probably not) or whether we want a dynamic 
webinterface provide this exact view (possible thanks to SRPM Id).

We could, for the time being, simply provide a web-interface that 
redirects to a (close) mirror on download. Which is what we do right now.

-- dag wieers, dag at wieers.com, http://dag.wieers.com/
-- dagit linux solutions, info at dagit.net, http://dagit.net/

[Any errors in spelling, tact or fact are transmission errors]

More information about the packagers mailing list