=====================================================================
   The PackageDB project has been retired in favor of the pkgdb2 project
        Confer: https://fedorahosted.org/pkgdb2/
   =====================================================================


================
Fedora PackageDB
================

:Author: Toshio Kuratomi
:Contact: [email protected]
:Date: Sun 7 October, 2012
:Version: 0.5.x

The `PackageDB` keeps track of package ownership within Fedora.

.. contents::

-------
Project
-------

The `PackageDB` was started in January of 2007 to remove some of the hurdles
that made it more time consuming to maintain packages in Fedora after CVS
acls were introduced.  The intial goal was to take information about who owned,
comaintained, and was allowed to commit changes to a package and put it in a
database where owners could edit it.

The system began to be used by cvsadmins in July of 2007, and was opened for
general use in September.  At present it allows general users to manage the
ownership of packages and allows admins to add new packages or change things
when the owner is unresponsive.  A webUI is open for everyone and a simplistic
commandline client works for admin requests.

.. _`PackageDB`: https://fedorahosted.org/packagedb

Future goals
============

We're trying to manage our Roadmap_ of new features in trac.  So far it's
working out well.  If you have a new feature request, please open a
`trac ticket`_ and we'll add it to the schedule to be worked on.

.. _`Roadmap`: https://fedorahosted.org/packagedb/roadmap
.. _`trac ticket`: https://fedorahosted.org/packagedb/report/3

------------
Installation
------------

Although it's not working yet, the goal is to be able to run::

  python seutp.py build
  python setup.py install --root ${path to installation}

and have a working installation.

Running from a Checkout
=======================

If you want to run from a checkout, you can perform the following commands::

Install Prerequisites
~~~~~~~~~~~~~~~~~~~~~
::
  yum -y install python-genshi TurboGears python-turbojson postgresql-server \
    postgresql-contrib python-TurboMail python-bugzilla python-fedora \
    python-psycopg2 python-sqlalchemy python-feedparser koji mod_wsgi \
    python-paver httpd mod_ssl babel python-cpio xz python-argparse \
    python-mako python-memcached

Get and Build the Source
~~~~~~~~~~~~~~~~~~~~~~~~
::
  git clone https://github.com/fedora-infra/packagedb.git
  cd packagedb
  python setup.py egg_info
  paver build --install-conf=`pwd` --install-data=`pwd` --install-sbin=`pwd`

Setup the Database
~~~~~~~~~~~~~~~~~~

.. note:: postgresql 8.3 or higher is needed

Make sure postgres is set to allow connections over tcp/ip using password
authentication.

If you do not have /var/lib/pgsql/data/pg_hba.conf, it may be because this is
a fresh install of postgres.  Initialize the service by entering the following
command::
  sudo service postgresql initdb

This should create several files in the /var/lib/pgsql/data directory including
the pg_hba.conf file.

Edit the /var/lib/pgsql/data/pg_hba.conf as the user postgres.  You might find
a line like this::

  host	all 	all	127.0.0.1/32	ident sameuser

Instead of that line, you need one that looks like this::

  host	all	all	127.0.0.1/32	md5

Create the postgres users that can access the database.  You will be prompted
to enter a password for each database user twice for verification purposes.
Note that the pkgdb web interface will use the pkgdbadmin user.  Other scripts
which only read from the database can use the pkgdbreadonly user to read
information only::

  sudo /etc/init.d/postgresql start
  sudo -u postgres createuser -SDRPE pkgdbadmin
    Enter password for new role: <password>
    Enter it again: <password>
  sudo -u postgres createuser -SDRPE pkgdbreadonly
    Enter password for new role: <password>
    Enter it again: <password>

Note:: If you don't have sudo, you can use su - instead::

  su -
  /etc/init.d/postgresql start
  su - postgres
  createuser -SDRPE pkgdbadmin
    Enter password for new role: <password>
    Enter it again: <password>
  createuser -SDRPE pkgdbreadonly
    Enter password for new role: <password>
    Enter it again: <password>

Get some test data and import it into the database.  You want to grab the most
recent dump from http://fedorahosted.org/releases/p/a/packagedb like this
(look for a .dump.xz that looks recent in that directory)::
  wget https://fedorahosted.org/releases/p/a/packagedb/pkgdb-20100318.dump.xz
  xzcat pkgdb-20100318.dump.xz | sudo -u postgres psql

This is a rather large dump and can take a while to load.

Update the Database Schema
--------------------------

Currently the database dump is not up to date with the latest changes in the
devel branch. These steps won't be required after we get a healthy 0.5.x dbdump.
Run the following scripts to bring it up to date::

  sudo -u postgres psql pkgdb < update-schema/01_static_icons.sql
  sudo -u postgres psql pkgdb < update-schema/02_timestamps_for_tags_and_usages.sql
  sudo -u postgres psql pkgdb < update-schema/03_add_gitbranchname.sql

Start memcached
---------------

Memcached is an optional daemon that can be run to speed up the server.  It
functions by caching certain data that is read many times so that we don't have
to access the database as frequently.  To use it you need to install and start it::

  su -c yum install memcached
  su -c /etc/init.d/memcached start

Setting up the cron jobs
------------------------
A cron job is needed to sync information from the yum repositories into the
packagedb. server-scripts/pkgdb-sync-yum update

For list of params use server-scripts/pkgdb-sync-yum update -h

Other cron jobs are required to sync data out to cvs, bugzilla, and koji
owner-sync-pkgdb
pkgdb-sync-bugzilla

We don't have a good way to test these.latter scripts.

TG FeedController bug
---------------------

We're using an old version of TG (1.0) that no longer gets any updates so
we'll have to modify the local TG everytime.

Apply this patch to turbogears/feeds/feed.py
http://trac.turbogears.org/attachment/ticket/2348/feedkid.diff

And here's the link to the upstream ticket:
http://trac.turbogears.org/ticket/2348

Note: this has been applied to the Fedora and EPEL packages.

App icons as static content
-------------------------------------------
Set server.allow_static_icons to enable static icons. This option influences 
the icon path generation as well as icon import part. This should make setup 
of already running server easier.

Set server.static_media_dir to directory which content is accessible from all 
pkgdb webservers. This path is used by import script as the target location 
of app icon tree. If server.static_media_dir is set to e.g. '/var/pkgdb/static'
'/var/pkgdb/static/appicon/' should be served as /pkgdb/static/appicon by apache.
This can be done by putting following line in webserver configuration:
    Alias /pkgdb/static/appicon /var/pkgdb/static/appicons

App icons tree is updated during regular build db update. Icons are collected in
DB first, when the import is finished the icon tree is updated with the new data.
During import the new icons are served from DB so that we can avoid 404s.

pkgdb-sync-yum has two new commands to admin static app icons:
 - icons2static (-all) will update (regenerate) the app icon tree
 - iconsfromdb will set all icons as unavailable on filesystem so that icons
   will be served from db


Start the Server (The new way)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

First copy the pkgdb.cfg.sample file to pkgdb.cfg.  Next, edit pkgdb.cfg to
configure the server to see your database.  You'll need to set fas.username to
a username (your Fedora Account System username will work), fas.password (if
you use your username, your password will work), and sqlalchemy.dburi.  The
dburi line will be something like this::

  sqlalchemy.dburi="postgres://pkgdbadmin:bunbunbun@localhost/pkgdb"

You'll need to change 'bunbunbun' to your password.

Next copy the httpd-pkgdb.conf file to your apache conf.d directory::

  sudo cp httpd-pkgdb.conf /etc/httpd/conf.d/

Restart apache::

  sudo /etc/init.d/httpd restart

In your favorite web browser, navigate to::
  https://localhost/pkgdb/


Start the Server (The old way)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. warning::

    This method of starting the server is deprecated.  It's helpful for
    quickly bringing up a test instance but using the wsgi server with apache
    is more scalable and how we run things in production.  This is currently
    working (with the 0.4.0 release) but there's not going to be much testing
    to make sure it continues to work.

First copy the pkgdb.cfg.sample file to pkgdb.cfg.  Next, edit pkgdb.cfg to
configure the server to see your database.  You'll need to set fas.username to
a username (your Fedora Account System username will work), fas.password (if
you use your username, your password will work), and sqlalchemy.dburi.  The
dburi line will be something like this::

  sqlalchemy.dburi="postgres://pkgdbadmin:bunbunbun@localhost/pkgdb"

You'll need to change 'bunbunbun' to your password.

Next you'll need a directory for the logs.  You have two options.  The first
is to create a directory in /var/log and change the owner and group to yours
as follows::
  sudo mkdir /var/log/pkgdb
  sudo chown <your username>:<your groupname> /var/log/pkgdb

Or you can edit pkgdb.cfg and change all references to /var/log/pkgdb to a
directory that you own.  For example: if you want to use ~/packagedb/logs,
then change the following lines::
  [[handlers]]
  [[[cron_out]]]
  args="('/var/log/pkgdb/cron.log', 'D', 7)"

  args="('/home/<username>/packagedb/logs/cron.log', 'D', 7)"


  [[[debug_out]]]
  class='TimedRotatingFileHandler'
  args="('/var/log/pkgdb/server.log', 'D', 7)"

  args="('/home/<username>/packagedb/logs/server.log', 'D', 7)"


  [[[access_out]]]
  class='TimedRotatingFileHandler'
  level='INFO'
  args="('/var/log/pkgdb/access.log', 'D', 7)"

  args="('/home/<username>/packagedb/logs/access.log', 'D', 7)"


  [[[error_out]]]
  class='TimedRotatingFileHandler'
  args="('/var/log/pkgdb/server.log', 'D', 7)"

  args="('/home/<username>/packagedb/logs/server.log', 'D', 7)"


Then start the server::

  python start-pkgdb


In your favorite web browser, navigate to::
  http://localhost:8086/pkgdb/

------------
Translations
------------

You will need babel, setuptools, and transifex-client to run these commands::
  yum install babel setuptools transifex-client

Much information about using transifex for translations can be found in the
`transifex user's guide`_.  The information in this section is largely from
experimenting with the information in the `transifex client documentation`_

.. _`transifex user's guide`: http://help.transifex.net/user-guide/index.html
.. ~`transifex client documentation`: http://help.transifex.net/user-guide/client/index.html

Updating the POT File
=====================

When you make changes that change the translatable strings in the package, you
should update the POT file.  Use the following distutils command (provided by
python-babel) to do that::
  ./setup.py extract_messages -o translations/pkgdb.pot
  tx push -s

Then commit your changes to source control.

Updating the PO Files
=====================

Transifex.net will merge the strings inside the pot file with the already
translated strings.  To merge these, we just need to pull revised versions of
the po files::
  tx pull -a

Then commit the changes to source control

Note that if you see a status message from transifex like this::
  Pulling new translations for resource fedora-packagedb.pkgdb.pot (source: translations/pkgdb.pot)
   -> fr: translations/fr.po

it means that transifex has created a brand new po file for you.  You need to
add the new file to source control before committing.

Creating a new PO File
======================

.. note:: translations are not yet setup in transifex.  This is due to the
    packagedb code not yet being setup to make use of translations.

The easiest way to create a new po file for a new language is in transifex's
web UI.

* Visit this link:
  https://www.transifex.net/projects/p/packagedb/resource/python-fedorapot/
* Login with a transifex.net username and password
* Scroll down to the add translation button below the statistics on current
  translations
* Select a language from the dropdown and hit the translate online button

This will start a translation for your language.  In the future you can
continue to use the transifex online translation editor or you can download
the .po file, translate on your computer, and then reupload.

Compiling Message Catalogs
==========================

Message catalogs can be compiled for testing and should always be compiled
just prior to release.  Do this with the following script::
  python releaseutils.py build_catalogs

Compiled message catalogs should not be committed to source control.

Installing Message Catalogs
===========================

``python releaseutils.py install_catalogs`` will install the catalogs.  This
commmand may be customized through the use of environment variables.  See
``python releaseutils.py ==help`` for details.

-------
Release
-------

0) Merge all features, hotfixes, etc that you want for this release into the
   develop branch

1) Checkout a copy of the repository and setup git flow::
     git clone [email protected]:fedora-infra/packagedb.git
     cd packagedb
     git flow init

2) Merge everything into a release branch:

     git flow release start $VERSION

3) Download new translations

.. note:: Not yet setup for translations.  Disregard this  section

     tx pull -a
     python releaseutils.py build_catalogs
     # If everything checks out
     git commit -m 'Merge new translations from transifex.net'

4) Make sure the NEWS file is accurate (use git log if needed).
   Commit everything to your branch

5) Update fedora-packagedb.spec and pkgdb/release.py with the new version
   information.
     # Make edits to fedora-packagedb.spec and release.py
     git commit

6) Make sure the docs are proper and publish them::
    # Build docs and check for errors
    python setup.py build_sphinx
    # packagedb isn't on pypi so don't use upload_docs.  Upload to
    # fedorahosted instead.  The directory to upload to is slightly different
    # depending on whether you're a fedorahosted admin or a normal
    # fedorahosted user:
    # Admin::
    rsync -av build/sphinx/html/ fedorahosted.org:/srv/web/releases/p/a/packagedb/doc/
    # Normal contributor::
    rsync -av build/sphinx/html/ fedorahosted.org:packagedb/doc/

7) Checkout a fresh copy of the release branch (so we know we aren't including
   any problematic files)::
    pushd ../
    git clone packagedb packagedb-release
    cd packagedb-release
    git checkout release/$VERSION

8) Build the tarball::
    python setup.py sdist --format bztar

9) Copy the dist/fedora-packagedb-VERSION.tar.bz2 and fedora-packagedb.spec files
   to where you build Fedora RPMS.  Do a test build::
     cp dist/fedora-packagedb-*.tar.bz2 fedora-packagedb.spec /srv/git/fedora-packagedb/
     pushd /srv/git/fedora-packagedb/
     rpmbuild -bs --nodeps fedora-packagedb.spec
     mock -r epel-6-i386 *.src.rpm

10) Make sure the build completes.  Run rpmlint on the results.  Install and
    test the new packages::
     rpmlint *rpm
     sudo rpm -Uvh *noarch.rpm
     [test]

11) When satisfied that the build works, create a fresh tarball and upload to
    fedorahosted.  The directory to upload to is slightly different for
    fedorahosted admins vs normal fedorahosted users:
    Admin::
     scp dist/fedora-packagedb*tar.bz2 fedorahosted.org:/srv/web/releases/p/a/packagedb/
    Normal contributor::
     scp dist/fedora-packagedb*tar.bz2 fedorahosted.org:packagedb/

12) Mark the release as finished in git::
     git flow release finish $VERSION
     git push --all
     git push --tags

13) Finish building and pushing packages for Fedora.