redist

2.0.0

An extensive system to manage and create quicklisp distributions.

About Redist

This is a system implementing facilities to produce Quicklisp distributions. Specifically, it implements the following:

  1. Scanning of ASD files to discover system definitions and dependencies, without having to load the ASD or any systems in question.

  2. A protocol to describe the objects related to dists, projects, and releases.

  3. A mechanism to serialise and restore these objects to retain the information across images.

  4. Integrations with source control systems to both clone and update project source files.

  5. Parsers for the quicklisp-projects information, making it possible to create a dist just like the official "quicklisp" dist.

  6. A compiler that takes the project source files and dist information to produce the tarballs and metadata files Quicklisp expects for a dist. Ultimately you can just point an HTTP server at the output directory and your dist setup is complete.

  7. A testing system to allow compiling each of the systems in a dist release, and verify that they still perform as expected.

Quickstart (Binary)

You can download a precompiled binary of Redist from the releases, or create it yourself by cloning Redist and using:

sbcl --eval '(asdf:make :redist)'

You should end up with a redist binary. When you invoke it, it should spill out a help that explains all the options and parameters. For a quickstart, first create your dist:

./redist add-dist my-dist --url "http://my.public.url"

Then create the projects you want to add:

./redist add https://github.com/someone/my-project.git
./redist add https://someone.com/project.tgz

This will create a sources/ directory where each project's source files are kept.

Finally you can compile the dist files:

./redist compile -v

Once your dist has been compiled, it'll have created a releases/ directory. You can simply copy this file to a static HTML server or serve them directly using nginx or similar.

In the future to create a new dist release, you can simply invoke:

./redist compile -vu

To update all the projects, first. The -v flag prints output. You can also add -j 4 or similar to speed up the process by running it on multiple threads.

Systemd Service

On most Linux systems you can set up a Systemd service file like /etc/systemd/system/redist.service

[Unit]
Description=Redist Distribution Compilation
After=network.target

[Service]
Type=oneshot
ExecStart=/home/redist/redist compile -vuj
WorkingDirectory=/home/redist
User=redist
Group=redist

Assuming there's a redist user that holds the redist binary and all the data. You can then trigger a recompile via systemctl start redist.

If you want to automatically rather than manually trigger compiles, you can also create a timer file like /etc/systemd/system/redist.timer

[Unit]
Description=Redist Releases

[Timer]
OnCalendar=monthly
Persistent=true

[Install]
WantedBy=timers.target

And enable it via systemctl enable redist.timer. Now an automatic release will be triggered every month.

Quickstart (Lisp)

Creating your own dist is rather simple:

(redist:define-project my-project
    ((:git "https://github.com/someone/my-project.git")))

(redist:define-project other-project
    ((:http "https://someone.com/project.tgz")))

(redist:define-dist my-dist (my-project other-project)
  :url "http://my.public.url")

This will create the dist object including projects and clone their sources to disk. If there's any problems with the cloning process, it will signal an error and allow you to continue in several ways.

Once a dist is set up, you'll want to serialise its current configuration to disk to ensure you can restore it at a later point when you want to update it.

(redist:store T T T)
(redist:retrieve T T T)

This will save all the information to disk in a distinfo.lisp file and an accompanying directory of bits. The directory structure is important to allow it to load information lazily as it's needed. Over time, the entire history of a dist is stored that way, which can grow quite large. Lazy loading is important to make sure startup remains snappy.

Once you have the dist set up with all the projects you'd like, it is time to make a release. To do so, simply call compile.

(redist:compile 'my-dist)

This will produce the necessary tarballs and metadata files. All you have to do now to make it accessible, is to point an HTTP server at the URL you declared earlier to the releases directory. Note that as of now, the Quicklisp client cannot talk HTTPS, so you must make your server accessible over plain HTTP.

Once a dist release has been produced, you should persist the changes, to ensure that when you create a new release it can share archives of projects that have not updated, and that it can properly regenerate the dist version index.

Should you want to add or remove projects from a dist later, you can simply redefine your dist, or use the functions add-project and remove-project, if you prefer. You can also maintain multiple dists with different sets of projects and different release schedules. The dists will share the archives whenever project versions coincide.

In order to update the project sources, you can either individually update a project, or pass :update T to compile. Additionally, if you'd prefer some extra output, since creating a release can take a while, you can also pass :verbose T.

If you would like to produce a dist from the official quicklisp-projects, simply clone that repository and load its data in via parse-quicklisp-projects. Be careful though, cloning all the repositories takes quite a while, and a few of the links have since broken.

HTML

Along with the actual dist files, Redist also generates a set of HTML. Specifically, it does so for:

  • The set of dists / projects you serve

  • Each dist

  • Each project

  • Each release

  • Each project release

This should help people explore the stuff you host on your servers without having to actually install the dist. It also helps to have a more interactive way to search through versions.

If you would like to customise how the pages look, please have a look at the files in the template/ directory. The HTML pages are generated using Clip, and the CSS is generated using LASS.

Programs

The following programs should be accessible for some features of Redist to operate:

  • curl
    For sources hosted as bare archives and replicating dists

  • bsdtar
    For sources hosted as bare archives and replicating dists

  • git
    For sources hosted in Git repositories

  • hg
    For sources hosted in Mercurial repositories

  • darcs
    For sources hosted in Darcs repositories

  • svn
    For sources hosted in Subversion repositories

  • cvs
    For sources hosted in CVS repositories

System Information

2.0.0
Yukari Hafner
zlib
https://Shirakumo.github.io/redist/
https://github.com/Shirakumo/redist/issues
https://github.com/Shirakumo/redist.git

Definition Index