From 23f0e1561767dd8a396188e317bae5920d171ea8 Mon Sep 17 00:00:00 2001 From: erdgeist Date: Sun, 16 Aug 2015 16:38:25 +0200 Subject: Initial import of my nikola website --- stories/projects/Telefonbuch.rst | 7 + stories/projects/anonbox.rst | 7 + stories/projects/briefkasten.rst | 6 + stories/projects/el.rst | 72 ++++++ stories/projects/elektropost.rst | 197 ++++++++++++++++ stories/projects/etherpad.rst | 237 +++++++++++++++++++ stories/projects/ezjail.conf.5 | 229 ++++++++++++++++++ stories/projects/ezjail.rst | 412 +++++++++++++++++++++++++++++++++ stories/projects/ezjail_philosophy.rst | 50 ++++ stories/projects/jaildaemon.rst | 77 ++++++ stories/projects/minimunin.rst | 75 ++++++ stories/projects/opentracker.rst | 136 +++++++++++ stories/projects/timestretch.rst | 63 +++++ stories/projects/vchat-client.rst | 267 +++++++++++++++++++++ 14 files changed, 1835 insertions(+) create mode 100644 stories/projects/Telefonbuch.rst create mode 100644 stories/projects/anonbox.rst create mode 100644 stories/projects/briefkasten.rst create mode 100644 stories/projects/el.rst create mode 100644 stories/projects/elektropost.rst create mode 100644 stories/projects/etherpad.rst create mode 100755 stories/projects/ezjail.conf.5 create mode 100644 stories/projects/ezjail.rst create mode 100644 stories/projects/ezjail_philosophy.rst create mode 100644 stories/projects/jaildaemon.rst create mode 100644 stories/projects/minimunin.rst create mode 100644 stories/projects/opentracker.rst create mode 100644 stories/projects/timestretch.rst create mode 100644 stories/projects/vchat-client.rst (limited to 'stories/projects') diff --git a/stories/projects/Telefonbuch.rst b/stories/projects/Telefonbuch.rst new file mode 100644 index 0000000..4c4260c --- /dev/null +++ b/stories/projects/Telefonbuch.rst @@ -0,0 +1,7 @@ +.. link: +.. description: is a project to quickly dump a certain kind of digital phone books. +.. tags: project +.. date: 2015/02/01 19:10:08 +.. title: Telefonbuch +.. slug: ../arts/software/Telefonbuch/index +.. prio: 98 diff --git a/stories/projects/anonbox.rst b/stories/projects/anonbox.rst new file mode 100644 index 0000000..404ddbb --- /dev/null +++ b/stories/projects/anonbox.rst @@ -0,0 +1,7 @@ +.. link: +.. description: is a web service to generate and maintain throw away email addresses, instanced on anonbox.net, including a javascript implementation of a unicode capable mbox-to-html renderer. +.. tags: project +.. date: 2015/02/01 19:10:08 +.. title: anonbox +.. slug: ../arts/software/anonbox/index +.. prio: 99 diff --git a/stories/projects/briefkasten.rst b/stories/projects/briefkasten.rst new file mode 100644 index 0000000..de4c8cd --- /dev/null +++ b/stories/projects/briefkasten.rst @@ -0,0 +1,6 @@ +.. link: +.. description: is a web service to anonymously submit messages via the web generating PGP mails and is instanced on the zeit-online briefkasten. +.. tags: project +.. date: 2015/02/01 19:10:08 +.. title: briefkasten +.. slug: ../arts/software/briefkasten/index diff --git a/stories/projects/el.rst b/stories/projects/el.rst new file mode 100644 index 0000000..8633181 --- /dev/null +++ b/stories/projects/el.rst @@ -0,0 +1,72 @@ +.. link: +.. description: is a unix tool aiding the Telefonbuch project for export and search queries. +.. tags: project +.. date: 2015/02/01 19:10:08 +.. title: el +.. subtitle: extract lines from text files +.. slug: ../arts/software/el/index +.. prio: 100 + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +Overview +======== + +*el* (from extract lines) is a unix tool to efficiently extract lines from a text file using a dynamic line end index. Its command line syntax is tuned to work with other posix tools – especially grep – to easily operate on poor man's databases where each of the database's columns is in a separate newline separated file. + +Since *el* mmaps the whole file, it does not work on streams. + +---- + +Build instructions +================== + +Get the source from my git repository ``git clone git://erdgeist.org/el`` and type ``make`` in the directory ``el``. *el* has no further dependencies and can just be copied to ``/usr/local/bin`` or wherever your binaries reside. + +---- + +Invocation +========== + +In its simplest form just run:: + + echo 1 2 3 4 5 6 7 8 9 10 | el file.txt + +to extract lines 1 to 10 from the file ``file.txt`` and print them to stdout. *el* also accepts white space separated line numbers with the ``-i`` parameter and can default to hexadecimal numbers with the ``-x`` switch:: + + el -i "10 11 12 13" -x file1.txt file2.txt + +extracts line 16 to 19 from the files ``file.txt`` and ``file2.txt``, each line separated by the tab character. If your line numbers start at ``0``, *el* will add 1, if it is run with the ``-0`` switch. + +The reason reading line numbers from stdin and not from command line by default becomes apparent, when you use the combining features of *el*, as it can parse and generate output as the ``grep`` posix tool does. + +If you use the ``-n`` switch, each extracted line is prefixed by it's line number and a colon, just as ``grep`` does. If you use the ``-g`` switch, *el* will also **expect** the line numbers on stdin to be terminated by a colon followed by garbage until next newline. So you can feed results from ``grep -n`` directly to it. With the ``-G`` option, not only will *el* parse the line numbers from ``grep``'s output, but also append the extracted lines from each file to the input line. In order to chain several ``grep`` and *el* commands, best combine the ``-Gn`` switches as in this example:: + + grep -n Manager positions.txt | el -Gn firstname.txt lastname.txt | grep Jon | el -Gn salary.txt + +This will produce, an output like this:: + + 17:Manager\tJon\tMiller\t$1200 + 23:Manager\tJon\tDoe\t$1600 + +given that all the text files contain corresponding rows of information about the staff. + +---- + +Author +====== + +*el* was written by `Dirk Engling `_, who likes to hear from happy customers. + +---- + +License +======= + +*el* is considered `beer ware `_. + + diff --git a/stories/projects/elektropost.rst b/stories/projects/elektropost.rst new file mode 100644 index 0000000..b8ad9d1 --- /dev/null +++ b/stories/projects/elektropost.rst @@ -0,0 +1,197 @@ +.. link: +.. description: is an ongoing mail server and webmail frontend project, documented to be set up on a FreeBSD jail and instanced on elektropost.org. +.. tags: project +.. date: 2014/02/01 19:10:08 +.. title: elektropost +.. subtitle: a FreeBSD jail based mail server project. +.. slug: ../arts/software/elektropost/index +.. prio: 750 + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +Overview +======== + +`Here is a raw sketch `_ of how the elektropost setup works. + +Send feedback to `erdgeist `_. + +---- + +Install qmail +============= + +* ``make patch`` in /usr/ports/mail/qmail-tls, choose SMTP_AUTH_PATCH, BIG_TODO_PATCH, BIG_CONCURRENCY_PATCH, DISCBOUNCES_PATCH and SPF_PATCH, keep RCDLINK +* apply patch http://erdgeist.org/arts/software/Code/elektropost/validrcptto.cdb.patch.new to qmail-tls/work/qmail-1.03 +* apply patch http://erdgeist.org/arts/software/Code/elektropost/qmail-smtpd.c.privacy.patch to qmail-tls/work/qmail-1.03 +* ``make install`` in /usr/ports/mail/qmail-tls +* add "QMAIL_SLAVEPORT=tls" in /etc/make.conf + +---- + +Configure qmail +=============== + +* In /var/qmail/control check/edit all files +* create a servercert.pem. On elektropost this one is used for lighty and dovecot as well, YMMV +* install http://erdgeist.org/arts/software/Code/elektropost/tcp.smtp to /etc/tcp.smtp and alter it accordingly to your needs, ie fix/insert to "cat /etc/tcp.smtp | tcprules /etc/tcp.smtp.cdb /etc/tcp.smtp.tmp" + + * for this you will need to ``make install`` in /usr/ports/sysutils/ucspi-tcp + * select SSL, leave MAN checked +* ``ln -s /var/qmail/boot/qmail-smtpd.rcNG /usr/local/etc/rc.d/qmail-smtpd`` +* ``ln -s /var/qmail/boot/maildir /usr/local/etc/rc.d/qmail`` +* ``echo 'qmailsmtpd_enable="YES"' >> /etc/rc.conf`` +* ``echo 'qmailsmtpd_checkpassword="/usr/local/vpopmail/bin/vchkpw"' >> /etc/rc.conf`` + +---- + +Install vpopmail +================ + +* ``make install`` in /usr/ports/mail/vpopmail +* Use default options for ucspi-tcp +* ``chown vpopmail:vchkpw /usr/local/vpopmail`` +* ``chmod u+s ~vpopmail/bin/vchkpw`` +* ``pw user mod vpopmail -s /bin/sh`` + +---- + +Configure vpopmail +================== + +* ``echo elektropost.org > /usr/local/vpopmail/etc/defaultdomain`` + +---- + +Install dovecot +=============== + +* ``make install`` in /usr/ports/mail/dovecot +* choose option VPOPMAIL, keep the rest (maybe disable IPV6) +* ``echo 'dovecot_enable="YES"' >> /etc/rc.conf`` + +---- + +Configure dovecot +================= + +* ``cp /usr/local/etc/dovecot-example.conf /usr/local/etc/dovecot.conf`` +* Change: protocols = imap imaps pop3s +* Change: listen = and ssl_listen = to match your ip addresses +* Change: ssl_cert_file = /var/qmail/control/servercert.pem +* Change: ssl_key_file = /var/qmail/control/servercert.pem +* Change: mail_location = maildir:%h/Maildir +* Change: first_valid_uid = 1 +* Change: first_valid_gid = 1 +* Change: max_mail_processes = 1024 +* Change: auth_username_format = %Ln@%Ld +* Change: #passdb pam { and #} if you do not have local user mail +* Change: #userdb passwd { and #} if you do not have local user mail +* Change: userdb vpopmail { and } to use vpopmail's vchkpw + +---- + +Install lighty +============== + +* ``make install`` in /usr/ports/www/lighttpd +* ``echo 'lighttpd_enable="YES"' >> /etc/rc.conf`` + +---- + +Configure lighty +================ + +* ``cp /usr/local/etc/lighttpd.conf.sample /usr/local/etc/lighttpd.conf`` +* Change: "mod_fastcgi", +* Change: server.document-root = "/usr/local/www/squirrelmail/" +* Change: server.port = 443 +* Change:: + + cgi.assign = ( ".pl" => "/usr/bin/perl", + ".cgi" => "/usr/bin/perl", + "/cgi-bin/qmailadmin" => "", + "/ezmlm-cgi" => "" ) + +* Change: ssl.engine = "enable" +* Change: ssl.pemfile = "/var/qmail/control/servercert.pem" +* To inform users to use https, put a document to /usr/local/www/data/index.html and add:: + + #compatibility + $SERVER["socket"] == ":80" { + ssl.engine = "disable" + server.name = "" + server.document-root = "/usr/local/www/data" + } + +---- + +Install squirrelmail +==================== +* ``make install`` in /usr/ports/mail/squirrelmail +* Use default options for php5 +* Follow the instructions squirrelmail port prints out + +---- + +Install qmailadmin / ezmlm-idx +============================== + +* ``make install WITH_SPAM_DETECTION=TRUE SPAM_COMMAND="| /usr/local/bin/spamc -f | /usr/local/bin/maildrop" CGIBINDIR=www/squirrelmail/cgi-bin CGIBINSUBDIR= WEBDATADIR=www/squirrelmail WEBDATASUBDIR=qmailadmin in /usr/ports/mail/qmailadmin`` +* Use default options for ezmlm-idx + +---- + +Install qmailadmin plugin for squirrelmail +========================================== + +* ``make install`` in /usr/ports/mail/squirrelmail-qmailadmin_login-plugin + +---- + +Install maildrop +================ + +* ``make install`` in /usr/ports/mail/maildrop + +---- + +Install the maildrop spam sort magic +==================================== + +* install http://erdgeist.org/arts/software/Code/elektropost/maildroprc to /usr/local/etc/maildroprc + +---- + +Install spamassassin +==================== + +* ``make install`` in /usr/ports/mail/p5-Mail-SpamAssassin, keep SPAMC, SSL, uncheck all others +* ``echo '-d 192.168.0.2' > /usr/local/etc/mail/spamassassin/spamc.conf`` + +---- + +Install the Valid RCPTTO checking stuff +======================================= + +* install http://erdgeist.org/arts/software/Code/elektropost/mkvalidrcptto to /usr/local/vpopmail/bin/mkvalidrcptto +* ``chmod +x /usr/local/vpopmail/bin/mkvalidrcptto`` +* ``cpan install CDB_File`` to help perl understand cdb files +* ``make install`` in /usr/ports/databases/cdb/ +* Add a script to your crontab (e.g. /usr/local/vpopmail/etc/tidymail.sh) that contains:: + + /usr/local/vpopmail/bin/mkvalidrcptto | /usr/local/bin/cdbmake-12 /var/qmail/control/validrcptto.cdb /tmp/validrcptto.tmp + +---- + +Install the auto trash-pruning script +===================================== + +* Add a script to your crontab (e.g. /usr/local/bin/prune-trash.sh) that contains:: + + find -E /usr/local/vpopmail/domains/ -type f -iregex "^.*/maildir/.*\.(junk|trash|spam)(\.[^/]+)?/(new|cur|tmp)/.*$" -mmin +86400 -delete + diff --git a/stories/projects/etherpad.rst b/stories/projects/etherpad.rst new file mode 100644 index 0000000..cc75def --- /dev/null +++ b/stories/projects/etherpad.rst @@ -0,0 +1,237 @@ +.. link: +.. description: is a documentation project on how to setup the classic etherpad software in a FreeBSD jail. +.. tags: project +.. date: 2015/02/01 19:10:08 +.. title: etherpad +.. slug: ../arts/software/etherpad/index + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +Overview +======== + +This is how to install the etherpad from http://etherpad.org/ in a FreeBSD-8.2 jail. (N.B: Things are reported not to work with newer versions.) + +This guide assumes, you have sudo, bash and an editor of your choice installed. Reserve around 2GB space for the jail. If you're using a different shell, use ``setenv`` to configure the environment variables set by ``export`` in this guide. Send feedback to `erdgeist `_. Many regards to `this guide `_, it helped me a lot. + +---- + +Install Scala + Java +==================== + +Since etherpad does not build under the version of scala that is shipped with FreeBSD-8.2 (i.e. scala-2.8.0), we have to manually install scala-2.7.7. However, scala brings nice dependencies that help with the manual installation of the jdk. So we install scala-2.8.0 to get the dependencies, then deinstall it and install java and scala. + +* Fetch jdk here: ``_ +* Fetch scala-2.7.7 here: ``_ + +Then type:: + + pkg_add -r scala + pkg_delete -f scala\* + pkg_add jdk-1.6.0.3p4_15.tbz scala-2.7.7.tbz + +---- + +Install mysql, fastjar and java connector +========================================= + +Type:: + + pkg_add -r mysql55-server mysql-connector-java fastjar + echo mysql_enable=YES >> /etc/rc.conf + /usr/local/etc/rc.d/mysql-server start + +---- + +Prepare mysql +============= + +Connection in a jail comes from the jail IP, not from 127.0.0.1, so enter IPADDRESS accordingly + +``mysql -u root`` + +and in the mysql shell:: + + create database etherpad; + grant all privileges on etherpad.* to 'etherpad'@'IPADDRESS' identified by 'MYSQLPASS_CHANGEME'; + +---- + +Checkout etherpad +================= + +Type:: + + pkg_add -r git + cd /usr/local + git clone git://github.com/ether/pad.git + +known good commit 94d5b2dfd65abbee5a06d80d94cdb00a7947d9b1, otoh: you might want to check `my issue about this commit `_, overlay windows are broken atm. + +---- + +Configure etherpad +================== + +Type:: + + cd /usr/local/pad/etherpad + cp etc/etherpad.localdev-default.properties etc/etherpad.local.properties + +and in your ``etc/etherpad.local.properties`` do: + +* Change: devMode = false +* Change: etherpad.isProduction = true +* Change: etherpad.proAccounts = true +* Change: etherpad.adminPass = ADMINPASS_CHANGEME +* Change: etherpad.isProduction = true +* Change: etherpad.SQL_PASSWORD = MYSQLPASS_CHANGEME +* Change: listen = 0.0.0.0:9000 +* Change: topdomains = pads.domain.tld,localhost +* Change: customEmailAddress = admin@pads.domain.tld +* Change: transportUseWildcardSubdomains = false +* Change: useHttpsUrls = true + +Remember the passwords well, take special care about trailing spaces! Now setup your domain in the following files: + +* src/main.js +* src/static/crossdomain.xml +* src/etherpad/pro/pro_utils.js +* src/etherpad/control/pad/pad_control.js +* src/etherpad/control/global_pro_account_control.js + +To disable a per-pad quota, comment out the call to _checkPadQuota in src/etherpad/control/pad/pad_control.js + +---- + +Rebuild the JAR +=============== + +Type:: + + export JAVA_HOME="/usr/local/jdk1.6.0/" + export JAVA="/usr/local/jdk1.6.0/bin/java" + export SCALA="/usr/local/share/scala-2.7.7/bin/scala" + export SCALA_HOME="/usr/local/share/scala-2.7.7/" + export SCALA_LIBRARY_JAR="/usr/local/share/scala-2.7.7/lib/scala-library.jar" + export PATH=/usr/local/share/scala-2.7.7/bin/:$PATH + export MYSQL_CONNECTOR_JAR="/usr/local/share/java/classes/mysql-connector-java.jar" + export mysql=mysql + +Rebuild the ace dependency +-------------------------- + +This is especially important if you see this (google hint): ``net.appjet.bodylock.JSRuntimeException: Error: Neither module exists: etherpad.collab.ace.easysync2, etherpad.collab.ace.easysync2.``:: + + cd /usr/local/pad/infrastructure/ace + bin/make normal etherpad + +Rebuild the JAR with correct settings +------------------------------------- + +Type:: + + cd /usr/local/pad/etherpad + bin/rebuildjar.sh + +---- + +Install nginx +============= + +Type:: + + cd /usr/ports/www/nginx + make install clean + +and be sure to select HTTPS. + +---- + +Create Self Signed Certificates +=============================== + +For whatever your domain is, substitute it in the following commands:: + + openssl genrsa -out pads.domain.tld.key 1024 + openssl req -new -key pads.domain.tld.key -out pads.domain.tld.csr + echo "subjectAltName=DNS:pads.domain.tld,DNS:*.pads.domain.tld" >pads.domain.tld.conf + openssl x509 -req -days 365 -in pads.domain.tld.csr -signkey pads.domain.tld.key -text -extfile pads.domain.tld.conf -out pads.domain.tld.crt + cat pads.domain.tld.crt pads.domain.tld.key >/usr/local/etc/nginx/pads.domain.tld.pem + +---- + +Configure nginx +=============== + +Nginx configs were provided by cryx. Thank you. Fetch ``nginx.conf`` here: ``_ and put it to ``/usr/local/etc/nginx/``. Then in your local ``nginx.conf`` do: + +* Change: server_name pads.domain.tld \*.pads.domain.tld; take special care to include the wildcard subdomains +* Change: ssl_certificate pads.domain.tld.pem; as created above + +Fetch ``nginx-etherpad.conf`` here: ``_ and put it to ``/usr/local/etc/nginx/``. Then in your local ``nginx-etherpad.conf`` do: + +* Change: server_name pads.domain.tld \*.pads.domain.tld; take special care to include the wildcard subdomains +* Change: proxy_pass http://pads.domain.tld:9000/; + +Type:: + + mkdir /var/log/www/; chown www:www /var/log/www + echo nginx_enable=YES >> /etc/rc.conf + +---- + +Install start script +==================== + +Fetch rc script here: ``_ and put it to ``/usr/local/etc/rc.d/``. + +Type:: + + chmod +x /usr/local/etc/rc.d/etherpad + echo etherpad_enable=YES >> /etc/rc.conf + mkdir /var/run/etherpad; chown etherpad /var/run/etherpad + chown -R etherpad:etherpad /usr/local/pad + +---- + +Install openoffice +================== + +If you want to export word and pdf documents, you need to install openoffice and run it as a headless service. + +Fetch the libreoffice package here: http://wiki.bsdforen.de/anwendungen/libreoffice_aus_inoffiziellen_paketen#freebsd_amd64i386_87-stable_kamikaze or, a know working mirror for ``_, MD5 (libreoffice-3.3.3.tbz) = 028d259735b361e46af8a9e253f168a8. + +Install dependencies, ignore warnings about already installed older versions:: + + pkg_add -r xineramaproto xf86vidmodeproto xextproto videoproto renderproto randrproto printproto kbproto inputproto fixesproto dri2proto damageproto compositeproto font-util encodings libtextcat expat gnome_subr ca_root_nss hicolor-icon-theme gnomehier lp_solve python27 xcb-proto perl hu-hunspell poppler-data png lcms jpeg jbigkit tiff openjpeg jasper curl pkg-config xproto pixman libtasn1 libfontenc libXdmcp libXau libICE libSM freetype2 mkfontscale mkfontdir fontconfig font-misc-meltho font-misc-ethiopic font-bh-ttf bitstream-vera xorg-fonts-truetype pcre orc m4 libpthread-stubs libxcb xcb-util libdrm libX11 libXt libXrender libXft libXfixes libXext libXxf86vm libXv libXrandr libXpm libXp libXmu libXinerama libXi libXdamage libXcursor libXcomposite libXaw libGL libGLU libltdl libffi libcheck icu db41 libiconv libxml2 raptor rasqal redland gettext libgpg-error libgcrypt glib gamin gio-fam-backend shared-mime-info libIDL gdk-pixbuf desktop-file-utils dbus libgee dbus-glib eggdbus cairo poppler bison gobject-introspection gstreamer gstreamer-plugins polkit pango ORBit2 lzo2 gnutls cups-client atk gtk dconf gconf2 libgsf libwpd libwps libwpg libxslt + +Install the local package:: + + pkg_add -f libreoffice-3.3.3.tbz + +Fetch rc script here: ``_ and put it to ``/usr/local/etc/rc.d/``. Then type:: + + chmod +x /usr/local/etc/rc.d/soffice + echo soffice_enable=YES >> /etc/rc.conf + +---- + +Configure name server +===================== + +Add records for the pads and wildcard subdomains, for tinydns this is:: + + +pads.domain.tld:192.168.0.23:43200 + +*.pads.domain.tld:192.168.0.23:43200 + +---- + +Filter local services +===================== + +Add rules to filter ports 9000 and 3306 to pf. diff --git a/stories/projects/ezjail.conf.5 b/stories/projects/ezjail.conf.5 new file mode 100755 index 0000000..f35d412 --- /dev/null +++ b/stories/projects/ezjail.conf.5 @@ -0,0 +1,229 @@ +.Dd December 5, 2013 +.Dt EZJAIL.CONF 5 USD +.Os FreeBSD +.Sh NAME +.Pa ezjail.conf +.Nd configuration file for ezjail script +.Sh DESCRIPTION +The file +.Pa ezjail.conf +contains settings that control the operation of the ezjail +.Xr rc 8 +script. It is also read by the +.Cm ezjail-admin +utility to figure out where it should perform its actions. Its path is +set at installation time to +.Pa EZJAIL_PREFIX/etc/ezjail.conf , +with an example file installed at +.Pa EZJAIL_PREFIX/etc/ezjail.conf.sample . +.Pp +This file is really a shell script that is sourced by the +.Cm ezjail-admin +command at run-time. +.Dq (str) +denotes a string; it should be enclosed in quotes if it contains space. +.Dq (bool) +notes a boolean, whose possible values are +.Dq YES +and +.Dq NO . +.Sh PATH OPTIONS +.Bl -tag -width option +.It ezjail_jaildir (str) +Location of jail root directories. +.br +Default: +.Em /usr/jails . +.It ezjail_jailtemplate (str) +Location of template jail used to create a new jail +.br +Default: +.Em ${ezjail_jaildir}/newjail . +.It ezjail_jailbase (str) +Location of base jail, the one that is mounted to all jails +.br +Default: +.Em ${ezjail_jaildir}/basejail . +.It ezjail_sourcetree (str) +Location of your copy of FreeBSD's source tree (refer to the +.Xr ezjail-admin 8 +utility for more information). +.br +Default: +.Em /usr/src . +.It ezjail_flavours_dir (str) +Location of the flavours, where each directory is a different flavour. +.br +Default: +.Em ${ezjail_jaildir}/flavours . +.It ezjail_ftphost (str) +This is where the install subcommand defaults to fetch its packages from. +.br +Default: +.Em ftp.freebsd.org . +.It ezjail_archivedir (str) +This is the default archive location for the +.Cm ezjail-admin archive +command. +.br +Default: +.Em ${ezjail_jaildir}/ezjail_archives . +.El +.Sh JAIL ADMIN OPTIONS +.Bl -tag -width option +.It ezjail_default_execute (str) +This is the default command executed in a jail by +.Cm ezjail-admin console . +.br +Default: +.Em /usr/bin/login -f root . +.It ezjail_exec_start (str) +The command to execute in a jail when starting it. +.br +Default: +.Em /bin/sh /etc/rc . +.El +.Sh JAIL CREATION OPTIONS +Default options for newly created jails. Used by the +.Xr ezjail-admin 1 +utility. Be careful about disabling +.Em ezjail_mount_enable . +.Bl -tag -width option +.It ezjail_mount_enable (bool) +Controls whether +.Pa /etc/fstab. Ar hostname +should be executed at jail startup time. +.br +Default: +.Em YES . +.It ezjail_devfs_enable (bool) +Controls whether newly created jails are given a working +.Pa /dev +directory. (Refer to +.Xr devfs 5 +and +.Xr jail 8 +for more information). +.br +Default: +.Em YES . +.It ezjail_devfs_ruleset (str) +Specifies which devfs ruleset should apply for newly created jails. +(Refer to +.Xr devfs 5 +and +.Xr jail 8 +for more information). +.br +Default: +.Em devfsrules_jail . +.It ezjail_procfs_enable (bool) +Controls whether newly created jails are given a working +.Pa /proc +directory. (Refer to +.Xr procfs 5 +and +.Xr jail (8) +for more information). +.br +Default: +.Em YES . +.It ezjail_fdescfs_enable (bool) +Controls whether newly created jails are given a working +.Pa /dev/fd/ +directory. (Refer to +.Xr fdescfs (5) +and +.Xr jail (8) +for more information). +.br +Default: +.Em YES . +.It ezjail_uglyperlhack (bool) +Set to YES, if ezjail should provide a soft link from +.Pa /usr/bin/perl +to +.Pa /usr/local/bin/perl +in base jail. +.br +Default: +.Em YES . +.It ezjail_default_flavour (str) +Controls which flavours should be used for newly created jails if none +are given on the command line. +.br +Default: +.Em none . +.It ezjail_imagetype (one of simple, bde, eli, zfs) +Type of jail to create when creating a jail with the +.Fl i +flag without specifying the type explicitly. +.br +Default: +.Em simple +.El +.Sh ZFS OPTIONS +.Bl -tag -width option +.It ezjail_use_zfs (bool) +Set to YES, if ezjail should manage basejail and newjail in a separate +ZFS-datasets. +.br +Default: +.Em NO . +.It ezjail_use_zfs_for_jails (bool) +Set to YES, if ezjail should manage all new jails in their own +ZFS-datasets. To override that on the command line, use +.Cm ezjail-admin create -c simple +for image based jails or +.Cm ezjail-admin create -c '' +for non-image jails. +.br +Default: +.Em NO . +.It ezjail_jailzfs (str) +The name of the parent ZFS-dataset which ezjail will use to create +jails on. It will be mounted in +.Em ezjail_jaildir . +Setting this will automatically enable ezjail managing jails in separate +ZFS-datasets. +.br +Default: +.Em none . +.It ezjail_zfs_properties (str) +Default properties ZFS will use for creating datasets. See +.Xr zfs 1m +for details. ADVANCED, be very careful! +.br +Default: +.Em none . +.It ezjail_default_retention_policy (str) +Policy for the +.Cm ezjail-admin snapshot +subcommand to keep older snapshots. See +.Xr ezjail-admin 1 +for details. +.br +Default: +.Em none . +.El +.Sh FILES +EZJAIL_PREFIX/etc/ezjail.conf +.br +EZJAIL_PREFIX/etc/rc.d/ezjail.sh +.Sh SEE ALSO +.Xr ezjail-admin 1 , +.Xr ezjail 5 , +.Xr jail 8 , +.Xr devfs 5 , +.Xr fdescfs 5 , +.Xr procfs 5 . +.Sh AUTHOR +Dirk Engling +.Aq erdgeist@erdgeist.org . +.Pp +The man page is based on a draft by +.An JoeB +.Aq joeb1@a1poweruser.com +and was rewritten by +.An Frederic Perrin +.Aq frederic.perrin@resel.fr . diff --git a/stories/projects/ezjail.rst b/stories/projects/ezjail.rst new file mode 100644 index 0000000..4eb4b65 --- /dev/null +++ b/stories/projects/ezjail.rst @@ -0,0 +1,412 @@ +.. link: +.. description: is a jail management framework for the FreeBSD operating system, aiming to aid in setting up und updating virtual FreeBSD instances. +.. tags: project +.. date: 2000/02/01 19:10:08 +.. title: ezjail +.. subtitle: Jail administration framework +.. slug: ../arts/software/ezjail/index +.. prio: 900 + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +News +==== + +.. image:: http://www.freebsd.org/gifs/daemon_hammer-tn25.jpg + :align: right + +.. post-list:: + :tags: ezjail + :template: list_projectnews.tmpl + :stop: 5 + +---- + +Overview +======== + +A `Jail `_ in FreeBSD-speak is one or more tasks with the same kernel Jail-ID, bound on zero or more IP addresses, having the same chroot-environment. One usecase of the FreeBSD Jail Subsystem is to provide virtual FreeBSD-systems within a Host-system. ezjail `is about making this as easy as possible `_, aiming for minimum system resource usage. All further references to the term Jail are to a virtual FreeBSD-system consisting of a host name, an IP-address and a Jail root. + +The jail(8) man page outlines the way to create Jails, however, when you need several Jails, complete Jail Directory Trees quickly use much of your valuable hard disc space. ezjail avoids this by using FreeBSDs nullfs feature. Most of the base system (/bin, /boot, /sbin, /lib, /libexec, /rescue, /usr/{bin, include, lib, libexec, ports, sbin, share, src}) only exists in one copy in the Host-system and is being mounted read only into all Jails via nullfs. Those Jails are quite slim (around 2mb each) and only consist of some soft links into the basejail mount point and non-shared directories like /etc, /usr/local, etc. + +The ezjail approach offers lots of advantages: + +* You save disc space, inodes and even memory since the system only needs to hold one copy of base system binaries for all Jails +* You can update all Jails on a single base directory, since it is so eazy, you might actually end up doing it +* Intruders compromising Jails are unable to install standard rootkits (as the base system is mounted read only) +* Since ezjail is written entirely in sh, there is no need to install other script languages into the hostsystem +* As the base system is provided via soft links, the enjailed users can choose not to use the mounted world +* ezjail offers full zfs integration and can help you automatize your file system configuration +* An often underestimated fact: less complexity means more security. + +---- + +Quick start +=========== + +To set up your first very simple ezjail, just install ezjail from ``sysutils/ezjail`` port or via ``pkg_add -r ezjail`` and enable it by setting ``ezjail_enable=YES`` in your in your ``/etc/rc.conf``. Assuming your network interface is em0, just type (as root):: + + ezjail-admin install + ezjail-admin create example.com 'em0|10.0.0.2' + ezjail-admin start example.com + +and you're done. Get a shell in your new jail with the:: + + ezjail-admin console example.com + +command. As with any vanilla FreeBSD installation, you might probably need to touch /etc/ and maybe copy your host's /etc/resolv.conf. + +---- + +Slow start +========== + +ezjail comes with some sane defaults, but can be configured globally and per jail using the config file /usr/local/etc/ezjail.conf (copy the sample from /usr/local/etc/ezjail.conf.sample) and the per-jail config files under /usr/local/etc/ezjail/ (those are created automatically with the jails and managed by the ezjail-admin config command). + +ZFS +--- + +ezjail integrates nicely with zfs, ready to manage all jails in its own file system. So if your system has a zpool configured, tell ezjail to use zfs and which zpool to use for its house keeping: + +* uncomment the ``ezjail_use_zfs=YES`` +* point the ``ezjail_jailzfs`` variable to a file system that will be created by ezjail-admin install, (e.g. ``tank/ezjail``) + +while you're at it, you can tell ezjail to create all future jails in their own file system (which defaults to be a child of ``ezjail_jailzfs``) + +* uncomment ``ezjail_use_zfs_for_jails=YES`` + +now the commands in the quick start example should set up a zfs hierarchy ready to use all the nifty features of zfs. + +Flavours +-------- + +ezjail can help you with the otherwise tedious task of decorating the interior of new jails–those come as naked FreeBSD installations by default. A set of files to copy, packages to install and scripts to execute is called "flavour". ezjail comes with an example flavour called "example" that comes pre-tuned for the use in jails, with an appropriate ``rc.conf``, ``make.conf``, ``periodic.conf`` and ``/usr/local/etc/sudoers``. + +You are encouraged to copy the flavour and modify the contained script to suit your needs–flavours reside in the directory configured with the ``ezjail_flavours_dir`` variable, which defaults to ``/usr/jails/flavours``. But just calling:: + + ezjail-admin create -f example example.com 'em0|10.0.0.2' + + +should do. Note, that the flavour script is being run the first time the jail starts, so calling:: + + ezjail-admin console -f example.com + + +is a nice idea. You can use the shell to further configure the new jail. + +The basejail +------------ + +All jails share a read only mounted copy of the FreeBSD base system, in ezjail this is called basejail. The quick start section gave a glimpse on the most simple way to install just the basics, but no ports tree, no man pages (pre FreeBSD-9) and no sources. You can run the ``ezjail-admin install`` command with the options -P, -M and -S again to install these distribution packages without installing the base system again, or just call ``ezjail-admin install -spm`` from start. ezjail uses the portsnap command to provide (and later update) the ports tree. If you do not want to install the OS version running in the host system, call:: + + ezjail-admin install -r 2.2.8-RELEASE + + +If you want to install your base system from source, use the ``ezjail-admin setup`` command (also called ``ezjail-admin update``); assuming you have already built your world for the host system, you would just call:: + + ezjail-admin setup -i + + +to run a make installworld from your source directory, which defaults to ``/usr/src``. To run a make buildworld before the installworld, call:: + + ezjail-admin setup -b + +For binary installations, ezjail uses the freebsd-update tool to keep the basejail up to date,:: + + ezjail-admin update -u + +should do the trick. + +Image and crypto jails +---------------------- + +Before the dawn of zfs, simple means to set limits on jails, like quotas, were hard to achieve. ezjail's answer were image jails, file backed "memory" disc images containing an ufs with the jail's content. When geom appeared with the very useful gbde and geli crypto layers for geom, encrypting image jails became possible. ezjail would handle creating and later attaching and detaching those images for you. + +Now simple image jails are not as hot anymore and personally I would recommend using geli to encrypt the provider for your zpool to apply proper crypto to all your jails. Still, there may be valid use cases for image jail. Call:: + + ezjail-admin create -i -s 2G example.com 10.0.0.2 + +to create a two gigabyte md-image with an ufs file system and install the jail inside. To configure the jail without starting it, use the attach and detach subcommands of ``ezjail-admin config``, like this:: + + ezjail-admin config -i attach example.com + cd /usr/jails/example.com + # … do your thing … + ezjail-admin config -i detach example.com + +Should the file system need some love, e.g. after a spontanous reboot or system crash, call:: + + ezjail-admin config -i fsck example.com + +to tidy up the mess–it ain't zfs, after all. By default ufs soft updates are enabled, so background fsck should occur for minor wrinkles, when an image jail starts. + +To create encrypted image jails, use the -c switch and either pass bde or eli and follow the instructions on screen:: + + ezjail-admin create -c eli -i 16G example.com 10.0.0.3 + +Note, that ezjail creates image jails by filling them from ``/dev/zero`` or ``/dev/random``, for performance reasons (reduce seeks with this file system inside a file system hack) and for security reasons (do not leak information about which blocks have been written for crypto jails), so creating huge image jails may take a while. Also note, that crypto jails would block the boot process (unless the passphrase is provided via a file or some fetch magic via stdin). So they are being marked as attachblocking and not started during boot time. You need to start them using ``ezjail-admin startcrypto``. + +Further reading +--------------- + +To get further details about all the options for ezjail-admin and all knobs and bolts, I recommend the excellent man pages: ezjail-admin, an overview over ezjail and the ezjail config file. + +---- + +Version History +=============== + +Since ezjail is useful under FreeBSD only, you might want to install it from ports ``sysutils/ezjail``. An `ezjail cvsweb (deprecated) `_ and `ezjail gitweb `_ are available, plus there is a `fresh checkout `_. The most current version is available via cvs. Use ``git clone git://erdgeist.org/ezjail`` or the legacy view ``cvs -d :pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co ezjail`` with an empty passwort to check it out. Just type ``sudo make install`` to install it. Older versions may be found here (checksums in tooltips): + +* CURRENT - not packaged yet +* V3.4.1 ezjail-3.4.1.tar.bz2 (2014-06-10) + + * Fixed a fatal bug in unfinished parts of the new troubleshoot subcommand + +* V3.4 ezjail-3.4.tar.bz2 (2014-06-10) + + * Began a troubleshoot subcommand that will in future check for unintuitive reasons for the jails to fail. + * Fixed a bug where Jails configured .norun were reported failing to start with the ezjail-admin start subcommand. Thanks to John Grasty. + * Create /var/ports/packages and /var/ports/distfiles in new jails to reflect custom /etc/make.conf. + * The Makefile now supports staging. + * Implement a test parameter for the ezjail-admin config -r subcommand that reports the runnable state. This can also be used to test for an ezjail's existence. + * Fixed a bug where ezjail-admin would not find jails with jid > 99999. Thanks to Mathieu Arnold. + * Fixed a bug where ezjail-admin delete and ezjail-admin config would not find the zpool the jail was created in. + * Implemented a jail snapshot mechanism for zfs jails with configurable retention policy. + * Fixed a bug where ezjail-admin restore with a path to an archive would not use the correct jail name and thus fail. Thanks to Ollivier Robert. +* V3.3 ezjail-3.3.tar.bz2 (2013-04-20) + + * Due to recent circumstances, I added a license file. + * Properly handle ip addresses with prefixed interface names, so they appear readable in the jail list and they will not be checked for on ezjail-admin create. Since FreeBSD 9.0 you can configure ip addresses with interface prefix (e.g. em1|10.0.0.1). This feature of jail(8) is now also documented in the ezjail's man pages. + * Using CVS to update ports was deprecated for a while. FreeBSD does not offer the anoncvs server, anymore. So ezjail_portscvsroot is gone. + * Added a -z option to ezjail-admin create to allow creating zfs jails in a different zpool or under a different zfs parent file system than the one ezjail uses for house keeping. Thanks to Cryx. + * Add the a man page authors section to the man pages. + * Also read and dump jail_JAIL_parameters when changing jail options. Thanks to Виталий Владимирович. + * ezjail-admin update and ezjail-admin upgrade now call freebsd-update install until no more update steps are necessary. Previously upgrades were not installed completely, as freebsd-update uses several passes. + * Now properly support the new distributions layout. + * Do not run chflags -R noschg when deleting image jails. The whole file system is being thrown away, anyway. Thanks to tomster. +* V3.2.3 ezjail-3.2.3.tar.bz2 (2012-09-23) + + * Fixed a bug that made ezjail-admin update not find the source tree. Thanks to Paul Macdonald. + * Fixed ezjail-admin console to not start all jails when the -f option is requested. +* V3.2.2 ezjail-3.2.2.tar.bz2 (2012-09-10) + + * Jails can now default to being created in zfs, set ezjail_use_zfs_for_jails="YES" in your ezjail.conf to do that. + * Cleaned up the zfs code to ensure the ezjail zfs in a central place. +* V3.2.1 ezjail-3.2.1.tar.bz2 (2012-02-02) + + * Fixed ezjail-admin list to work properly without jails. + * A helper script ezjail-clone.sh has been added to the repository that aids in importing whole server setups into an ezjail-system. + * ezjail-admin update now ensures that all files with schg set are also updated. Thanks to Geoffroy Desvernay. + * Fixed the way ezjail-admin create tests for local IP addresses. The old test using ping with a given TTL did not work reliable. Now we use ping and try to use the IP address as source address. + * Fixed a bug where pids not belonging to a Jail were reported as jailed. Thanks to Thomas Steen Rasmussen. + * Fixed a bug in ezjail-admin delete -f where all Jails were stopped, not only the one to be deleted. Tanks to Thomas Steen Rasmussen. + * Fixed a bug in ezjail-admin create that overwrote all flavours scripts with the successor by not counting the flavours correctly. Thanks to Geoffroy Desvernay. +* V3.2 ezjail-3.2.tar.bz2 (2012-01-07) + + * The man pages have been completely rewritten and are much easier to understand now. + * Default archive location now is ezjail_archives under ezjail_jaildir. Before it defaulted to the current directory. + * ezjail-admin install now understands FreeBSD's new naming scheme on the release ftp servers. + * /usr/games/ has gone. + * ezjail-admin delete now takes an -f parameter that forces stopping the Jail before deleting it + * The rc.d/ezjail.sh script has been renamed to rc.d/ezjail in accordance to FreeBSD guide lines. + * When using ezjail-admin to start Jails, checks now are properly extended to all versions of the (one|force|...) prefixes. + * When installing the basejail from a server, we now also look in the ftp server's Archive directory. + * The check for already listening services has been augmented to support multiple IPs and v6 addresses. + * Before removing Jails the system immutable flag is now cleared. + * ezjail-admin now collects its Jail list more to the standard. + * Unknow Jail image types now default to directory based. + * Experimental freebsd-upgrade support. + * Fix one misspelling that wrongly reported success for attaching a console to a Jail not running. + * The ezjail rc script now correctly also re-attaches images on restart. + * There now is a Jail name auto completion script for zsh. +* V3.1 ezjail-3.1.tar.bz2 (2010-02-26) + + * Introduced support for creating ezjails and the basejail in ZFS file systems. + * Attaching ZFS-datasets to ezjails is now supported. + * Support for cpusets and altered network view for ezjails. + * The ezjail-admin archive command now uses the cpio format for its archives. + * Since FreeBSD 8 does not support rc.d-scripts ending in .sh, the flavourizing script has been renamed accordingly. + * The ezjail-admin list command now supports multiple IP addresses for a jail. + * The ezjail-admin command now returns 0 if no error occurs. + * You can now specify a default flavour in your ezjail.conf that is applied to any new Jail. + * Renamed the default flavour to example to avoid confusion. default flavour was never applied by default. + * ezjail-admin create and ezjail-admin config no longer fail if they can't create a soft link in ezjail_jaildir. They now use mktemp to get a unique one. + * ezjail-admin create now allow multiple flavours to be applied to new jails. They are executed in the order provided with the -f option. +* V3.0 ezjail-3.0.tar.bz2 (2008-09-26) + + * Introduced an ezjail-admin restore command that can be used to easily deploy a backed up version of an ezjail's archive. + * Introduced an ezjail-admin archive command that – accompanied by the newly introduced create -a option – can be used to take (automated) backups of your jails and reinstall them. + * Introduced the ezjail-admin console sub command. This will jexec into a Jail identified by its Jail name. It can also bring up non-running jails if run with the -f switch. + * ezjail-admin update does not (re)build the whole world by default, anymore. You have to provide -b (buildworld, installworld), -i (installworld only) or (as a new option) -u to use freebsd-update to update your base jail. + * ezjail-admin now silently proxies everthing that looks like it was meant for ezjail.sh over to ezjail.sh. + * ezjail-admin now reports its version number along with synopsis information. + * ezjail.sh now stops ezjails in reverse startup order. + * ezjail.sh now reports ezjails configured norun as "skipping [jailname]" and blocking as "skipping blocking [jailname]". + * ezjail-admin now find .s its stuff. Using '*' never found dot-files and could have led to a "too many parameters" situation. + * Fixed a bug where image jails would not be run with ezjail.sh restart when not running or would not have their images attached when already running. Thanks to hukl for bug report. + * Fixed the missing lib32 packag on amd64 platforms when doing ezjail-admin install. + * sudo now checks permissions on its sudoers file and stops, if it is not 0440. Fixed ezjails Makefile to install sudoers to default flavour. (However, if you already installed ezjail, you might want to manually chmod 0440 ${ezjail_flavours}/default/usr/local/etc/sudoers and all corresponding sudoers files in all jails.) + * Fixed a bug, where an image Jail was incorrectly reported not attached. + +.. class:: closed + +* V2.1. ezjail-2.1.tar.bz2 (2007-07-31) + + * Introduced a ezjail-admin config -n newname sub command to enable renaming jails. + * Introduced a ezjail-admin config -i fsck sub command to fsck images. + * Use the sendmail_submit_enable, sendmail_outbound_enable and sendmail_msp_queue_enable in the default-flavour rc.conf to allow easier fine-grained control for sendmail. + * ezjail-admin create now ignores .dot-files in directories meant to be Jail roots. That way directories coming fresh from newfs won't be mistaken as used. + * Fixed a bug where portsnap would be called with fetch rather than cron from my ezjail-admin update -P. + * Enable soft updates for image jails. + * Now taking copies of soft linked packages inside jails. They would not be visible. + * Fail, when trying to install or update ezjails world, if in a secure level > 0. Too many bad things happen in secure levels. + * ezjail.sh now also checks for root directory to prevent strange effects in /etc/rc.d/jail. + * Default flavours /etc/periodic.conf now makes all periodic scripts log to files instead of sending mails. Most often this is what you want in your jails. + * Started learning groff and went over man pages. Not many visible changes here. +* V2.0.1 ezjail-2.0.1.tar.bz2 (2006-07-24) + + * DESTDIR was ignored for sub targets in an ezjail-admin update -p under certain circumstances. See this report for more details. Thanks go to Simon L. Nielsen. + * Fixed some qoutation/expansion issues with tr parameters in the form [:alnum:] which were expanded to the files : a l n u and m, if existing. Also an apparent bug in tr was circumvented which incorrectly replaced A-Z to a-z in certain LANG settings. Thanks agains to Simon L. Nielsen. + * ezjail_ftphost was ignored when specified in PREFIX/etc/ezjail.conf. Thanks to Edwin Groothuis. +* V2.0 ezjail-2.0.tar.bz2 (2006-05-31) + + * The default way to create the basejail will become the newly introduced ezjail-admin install subcommand. In its default configuration the base system is fetched from an ftp-Server. Fetching ports, source, man pages is possible with -p, -s and -m switch. Implementing this feature finally brings the promised ease of use and allows: cd /usr/ports/sysutils/ezjail; make install; ezjail-admin install; ezjail-admin create test.com 10.1.1.1 on a vanilla FreeBSD server, i.e. without installing /usr/src. + * A new concept of how to manage your Jails has been introduced: Image Jails. They come in plain, gbde and geli encrypted. ezjail-admin create, ezjail-admin delete and ezjail.sh have been updated to provide the functionality needed. + * ezjail.sh now provides more information to rcorder which took control over the execution of new style localpkg scripts. These changes in FreeBSD rc scripts led to ezjail.sh not being started under certain circumstances. It will REQUIRE: sshd, making administration on a server running many jails a lot more peaceful. Thanks to Oli. + * When called without specific Jail names (e.g. at system start), ezjail.sh now runs rcorder on its config file list located in PREFIX/etc/ezjail/ before processing them. That way you can manually construct a dependency tree and for example start your name server Jail first and your data base-Jail before your web server Jail. However, some basic knowledge in bourne shell and the rcorder command is required. Beware of circles. + * ezjail-admin create now installs flavours with an unconditional flag. Previously files were not copied, if an "older" version existed. Thanks to Matthias Lederhofer. + * ezjail-admin list has been heavily extended. It now lists Jails in the same order they are being run on system startup. This is to help debugging the dependency tree. It further now also lists jails status (Directory based, Image, BDE crypto image or ELI crypto image) and running state (Running, Attached, Stopped or Norun). Thanks to Bas Dakkenhorst for useful diffs. + * ezjail-admin create now prepends all Jail configs with empty rcorder control statements to make altering the config easier. + * ezjail.sh now excludes Jail configs whose file name contain '.' (dots). This way it is possible to configure a Jail config test_org.norun to temporary exclude it from auto run process. + * ezjail introduces an option ezjail_uglyperlhack (defaults to YES) which creates a soft link from /usr/bin/perl to /usr/local/bin/perl in the base jail. This is useful since perl was expelled from base system in FreeBSD 5 but still many scripts rely on #!/usr/bin/perl. + * /usr/games and /usr/libdata have now become part of the basejail. Someone please kick me for missing that until now. Saves another 100k in newjail. Hopefully I got them all by now ;) Thanks to Vivek Khera. +* V1.3.1 ezjail-1.3.1.tar.bz2 (2006-03-13) + + * ezjail.sh did not ignore the jail_list from rc.conf as it should due to a legacy name space clash. This led to no ezjail being started at system startup when some non-ezjail-Jails exist in the system. Thanks to Cryx. + * ezjail-admin now prevents a Jail called fulljail to be created and refines the error message in case the user wants to create basejail, newjail or flavours. + * ezjail-admin now again allows recreating Jails with the -x option. This was broken when introducing the Jail root check in V 1.3. Thanks to Mike Harding. + * An experimental script ezjail-release.sh has been added to CVS which allows creating the basejail from FreeBSD ftp-Server. However, this script has known issues, like updating existing basejail's ports does not remove obsolete files, also there is no documentation. +* V1.3 ezjail-1.3.tar.bz2 (2006-02-16) + + * Removed an ugly shell variable quoting bug that led to ezjail.sh fail for multiple jails. Thanks to Alex Moura. PLEASE UPDATE FROM 1.2 as fast as possible. + * ezjail-admin create now checks, whether a given Jail root already exists and refuses to install. Thanks to Mark Bucciarelli for finding this one. + * /boot and /rescue have now become part of the basejail. Someone please kick me for missing that until now. New jails now are < 2 MB in size. +* V1.2 ezjail-1.2.tar.bz2 (2006-02-10) + + * A subsystem providing auto configuration on Jail has been created. You may specify flavours at Jail creation. See section Flavours. + * ezjail-admin update now allows cvs co or cvs up of ports in the basejail by providing the -p or -P option. + * /lib and /libexec have now become part of the basejail. Someone please kick me for missing that until now. + * ezjail-admin now allows setup as an alias command to update to reduce confusion. + * ezjail-admin delete now ensures, that the Jail is not running. Deleting running Jails led to panics in several occasions. + * ezjail-admin update now performs several checks to make sure the source tree is really there and complete. + * ezjail-admin create now tests for services listening on the new Jails IP or on all IPs and warns if it finds any. + * ezjail-admin create now replaces any character in Jail name not suitable for shell variables by underscore. This leads to the situation that two jails must not have identical names after them being piped through tr -c [:alnum:] _, thanks go to Gunjin. + * A default installation PREFIX is provided now. Installing to / seems not too useful and should not be default. + * Documentation has been rewritten and simplified a lot. +* V1.1 ezjail-1.1.tar.bz2 (2005-10-26) + + * minor typo in ezjail.sh, changed #/bin/sh to #!/bin/sh, thanks go to Alex Samorukov. + * Makefile ensures that all directories are there when installing. Helps with odd $PREFIXes + * ezjail-admin now creates $basejail/usr itself, since cpio creates missing intermediate directories in 0700 which is bad + * ezjail-admin now verifies the existence of its basejail before creating Jails +* V1.0 ezjail-1.0.tar.bz2 (2005-10-14) + + * first release + * made FreeBSD port for ezjail + * actually implemented ezjail-admin delete feature + * documentation completed + * lots of bug fixes I can't remember anymore, however V0.1 has never been used by anyone but me +* V0.1 ezjail-0.1.tar.bz2 (2005-09-26) + + * highly experimental version + * first real world testing on a server system with >20 jails +* V0.0 - Not packaged (2005-09-04) + + * first alpha version + +FAQ +=== + +Q: I have created a Jail but it won't come up when I run /usr/local/etc/rc.d/ezjail start JAILNAME + A: Please check, that the Jail is really "not comming up" by examining the output of the command jls. Unlike the command jail which leaves you with an interactive shell in your Jail, rc.d/ezjail is an rc-script. It is most commonly meant to run at system startup and shutdown where entering an interactive shell would interrupt the boot process. If you want an interactive shell in your Jail after starting it, use the command ezjail-admin console JAILNAME. + + A: Please check, that you have enabled ezjail by adding the line ezjail_enable="YES" to rc.conf. + + A: Please check, that you have enabled ezjail BEFORE you ran the rc.d/ezjail script the first time. See the rc.d introduction for further details. + + A: If this is the case, check the console output in /var/log/jail_JAILNAME_console.log. Most commonly you'll find reasons for your jails startup failure there. + +Q: Why does ezjail want an extra copy of ports tree for its base jail? Isn't that just waste of disc space? Can't we just use the base systems ports tree? + A: Counter question: why do we need an extra copy of the base system? Isn't that just waste of disc space? Can't we just mount / as basejail to all jails? + + ezjail philosophy very much is influenced by its history: it was created to simplify administration of a shell server with many actual users. Even if you would trust all your users, you may not trust all software they do install and hence not anything done in any Jail. So I decided to strictly seperate what you can access within Jails from everything the host system needs to operate. This boundary is ezjail_root. + + For the ports case I would not like everyone to know which port is installed in host system, as might be guessed by work/-directories, this might reveal attack vectors to the host system. A problem like ignoring ro flags when mounting from fstab may give you an idea why I consider that boundary essential to system security. + + If you know what you're doing and are sure, your host system has an WRKDIRPREFIX none of the Jails can access, you might follow these instructions to mount ports via nullfs. Another way of doing this is to mv /usr/ports/ /usr/jails/basejail/usr/ports/; ln -s /usr/jails/basejail/usr/ports /usr/ports; in host system. ezjail will not offer this as an option. + +Q: What exactly is the difference between the template Jail and a Flavour? + A: In simple words: THE template is all directories that FreeBSD thinks your system needs besides the sharable files in your basejail, like /etc/ and /var/. It is being recreated everytime you ezjail-admin update. So it is not meant to be modified by the user to distribute settings to all Jails to be created. A Flavour on the other hand is set up by the user, contains all non-sharable files not in the base system that you think belongs to a newly created Jail (usually stuff in /usr/local/ and your settings in /etc/). You may have several Flavours each customized for a certain job: an httpd-Flavour, a shell-server only Flavour, a full featured bamp-server and so on. Flavours are never modified by ezjail, the template should not be modified by the user. + +Q: I am running version x of ezjail and want to upgrade to version y. Is there a safe upgrade path without losing all of my Jails? + A: ezjail is basically just two scripts. The config files have been designed to be future compatible. So installing any version from ports, cvs or tar ball by ezjails Makefile will do the job of upgrading. Your Jails are safe. + +Q: I have created some jails prior to using ezjail. How can I import them as ezjails? + A: There now is an import helper script in the ezjail tar bundle that assist you with moving complete systems into ezjails by turning them into ezjail-admin restore-able ezjail archives. + + A: Assume your Jail is under /legacyjail Basically you only need to create an empty directory /legacyjail/basejail/ and call ezjail-admin create -x -r /legacyjail JAILNAME JAILIP and your Jail enjoys all of ezjails starting/stopping capabilities.. + + To make the Jail a complete ezjail, e.g. use the basejail/, use the following code snippet after stopping the Jail. (Replace /legacyjail with the path to your Jail). + + .. code-block:: sh + + cd /legacyjail + mkdir OLDBASE + for dir in bin boot lib libexec rescue sbin usr/bin usr/include usr/lib usr/libexec usr/ports usr/sbin usr/src usr/share; do + mv ${dir} OLDBASE/; ln -s /basejail/${dir} ${dir} + done + + There is a chance that your binaries require some of the libraries in its old base system. If they don't, you can remove OLDBASE, if they do, use the following snippet to create the old libs. + +Q: The clock inside my Jails always shows the wrong time. Why is that? + A: Create a soft link from /etc/localtime to the appropriate time zone file, in my case: /usr/share/zoneinfo/Europe/Berlin. + +Q: I messed up the whole show and ezjails files are around EVERYWHERE. Some of them wont even go away, if I rm -rf them as root. How do I get a clean start? + A: On a standard FreeBSD installation several files are being installed with the "system immutable" flag set. Type man chflags for more details. However, to reset everything to a state with no jails and no files lurking around use the following lines, on kern secure level 0, as root, on a standard ``ezjail.conf`` + + .. code-block:: sh + + chflags -R noschg /usr/jails/ + rm -rf /usr/jails/ + rm -rf /usr/local/etc/ezjail/ + rm /etc/fstab.* + + If you configured ezjail to use zfs, you can just throw away ezjail's parent zfs, as configured in ezjail_jailzfs. + +Q: Your HTML sucks. + A: This is not a question. But yeah, I know. + +---- + +Author/Contact +============== + +ezjail was written by Dirk Engling with lot of support from the ezjail community. We like to hear from happy customers. My personal address is erdgeist@erdgeist.org. Requests should go to the project mailing list ezjail@erdgeist.org. You can subscribe to that list at ezjail-subscribe@erdgeist.org. There is an IRC channel #ezjail on ircnet. Please send bug reports, comments and feature requests. Donations are welcome… in form of beer. See… + +---- + +License +======= + +ezjail is considered `beer ware `_. + +---- + +Thanks +====== + +Here I need to re-add a modernized list of thanks. diff --git a/stories/projects/ezjail_philosophy.rst b/stories/projects/ezjail_philosophy.rst new file mode 100644 index 0000000..52e2f7f --- /dev/null +++ b/stories/projects/ezjail_philosophy.rst @@ -0,0 +1,50 @@ +.. link: +.. description: the philosophy behind the ezjail framework +.. tags: +.. date: 2015/02/01 19:10:08 +.. title: ezjail - philosophy +.. slug: ../arts/software/ezjail/philosophy + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +Overview +======== + +Computer programs tend to contain programming errors. Chances to avoid them in every program that provides services to the internet are low. What is needed is a way of limiting the impact of programming errors to the system hosting the service. + +Computers become faster and their resources in terms of memory, mass storage and internet connectivity can suite several users at once. This led to the development of time sharing computing, most popular in Unix. The next level of abstraction is virtual servers enabling every owner of such an entity to create users and use the power of multitasking as well, without even noticing that the machine is not completely under his control. + +Some server software expects its configuration to be always in the same location in file system and makes changing this location difficult to impossible. Some server software requires certain privileges you would not want to grant it, even if you would assume it being error-free. Some users want certain privileges you would not want to grant them, even if you would assume they know what they are doing. What is needed is a way of limiting the impact of permissions you have to grant. + +Historical approaches +===================== + +Several approaches to achieve most of the above have been made. It has shown that the most secure way to do so is to physically separate services to different machines. Of course, this has its costs, is a lot more complex since you have to reproduce and administrate the setup over and over. And, of course, it contradicts the idea of using the grown performance as shown in the second paragraph. Luckily most of the problems were solvable, solutions being condensed in the ezjail script set. + +FreeBSD has introduced the concept of jails to eliminate shortcomings seen in so called chroot-environments. chroot changes the file system root for certain programs, leaving them without any means of even expressing the need for access to resources outside their constricted world. Jails take this approach a little further, they represent virtual computers (virtual server), each bound to a single internet address having its file system root somewhere under the real computers (hosting server) file system root. Certain rulesets forbid jailed programs access to specific devices, for example hard drives or kernel memory. + +In theory this concept enables you to lock in each (server-)program you run on your computer to its own virtual world limiting the impact it may (deliberately or by mistake) have on other services. In practice each program depends on a set of libraries, tools, configuration directories, system users, other installed software and so on. In short: most servers need at least a small subset of the host servers operating system (OS). Complete virtual servers, of course, need a whole copy of the host servers OS. + +Problems in historical approaches +================================= + +Locking in a single program that way may work out fine, as it "only" doubles the size your operating system needs on hard disc, and hard disc space is becoming cheaper as I write, the FreeBSD OS only needs around 100MB. However, copying all the data does not scale well when you need hundreds of programs being jailed or want to provide dozens of virtual computers. Further, distributing OS software updates to all this instances, which possibly even were created from different versions of the OS source code, becomes an administration nightmare. + +Although hard disc space is becoming less of an issue, OSs tend to assume the space rather being occupied by few large files then many small files. So they reserve a limited number of control structures for storing files and directories, called "inodes". Now, copying the whole OS many times consumes many of those valuable inodes on disc, eventually preventing the creation of new virtual servers. Worse: since each library, each executable and each configuration file of the OS exists as several copies, the OS has no way of telling that they're the same and thus has to load each copy to memory, if any of the virtual servers or jailed programs need them. + +To complete the picture of the situation I was confronted with when starting the ezjail project: the way to install a copy of the OS to create a virtual server involved (and still involves) installing a copy of the OS source code and invoking several commands to compile, install and configure it. This still left the administrator with the need to start the virtual server by hand and entering a lot of complicated configuration statements to the OSs startup scripts in order to have them run automatically when the server starts up. Jailing programs or virtual servers was rather complicated, achieving any of the goals set at the beginning of this text far away. + +Developing a solution +===================== + +It was very helpful that Unix' file system hierarchy reflects historic optimizations. The `Filesystem Hierarchy Standard `_ defines a set of "sharable files", mostly OS binaries, libraries, header files and resources. Those directories are meant to be shared by several computers (via Net-boot, eventually read only from CD-ROM). "unsharable files" on the other hand, contain configuration data, dynamic data like user mails, installed applications etc. While the former directories only need to exist once for all jails and can be provided in a read only fashion, the latter must be provided for each jail and, of course, be writable. + +And here a path to the solution shows: most modern OSs know a way of "local mount"/"loopback mount"/"null mount" directories back into the directory hierarchy. That way it is possible to provide all sharable files in a read only fashion to the chroot environment of jailed programs. All unique data a virtual server need is in the "unshared files" which must be copied for each of them. Fortunately, these files are just 2MB in size on the FreeBSD OS. How the sharable files are being provided is subject to implementation. The ezjail project mounts one directory containing all those shareable files and provides soft links from all directories not in the unshared files part of the OS. + +Besides having a central point for OS updating all instances at once, this approach can prevent standard root kits to install, since most of the binaries they usually replace are in the unshared files, yet, when someone really wants to install their own version of the OS binaries, one can do so by just removing the soft links and provide self compiled system binaries. System administrators know exactly, which OS version problems in virtual servers are related to, since all virtual servers have the same base system. + +Everything left to do is to provide a neat interface, do some sanity checks, wrap it properly around the existing scripts of your OS and find a nice name. In my case: `ezjail `_. diff --git a/stories/projects/jaildaemon.rst b/stories/projects/jaildaemon.rst new file mode 100644 index 0000000..cc818e3 --- /dev/null +++ b/stories/projects/jaildaemon.rst @@ -0,0 +1,77 @@ +.. description: is a tool for the FreeBSD operating system to allow flexible and secure communication from jail environments to the host system. +.. tags: project +.. date: 2015/02/18 19:10:08 +.. title: jaildaemon +.. subtitle: A fixed command-line interface for FreeBSD jails to their host system +.. slug: ../arts/software/jaildaemon/index +.. prio: 850 + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +Overview +======== + +.. image:: http://www.freebsd.org/gifs/daemon_hammer-tn25.jpg + :align: right + +Virtual FreeBSD jail based server environments can not easily trigger actions in the host system, most prominently restarting themselves (if they ``kill -1`` everything, they usually stay shut down) and trigger actions like routing table updates, triggering zfs snapshots or rollbacks and configuring ``tun/tap``-Devices (as an openvpnd would). + +There are several less-than-optimal solutions for that problem: granting forced-command ssh access to jails, polling some Jail-state from within the host system or dig some proprietary socket holes through the Jail's barrier, exposing a root-shell attack vector to the outside world. + +*jaildaemon* aims to solve the problem in a simple and straight forward way while trying to minimize the attack surface for adversaries from the Jail and the network. It is a tiny daemon running in the host system that forks a probe process with a single associated command on demand, attaches it to a single jail and (optionally) assigns it a verbose proctitle. When the Jail's root user sends a SIGHUP to this probe process, it dies with a magic exit code, signalling the daemon to execute the associated command (in host context). You can chose to have the daemon restart the probe process after the command is executed (e.g. when it triggers updating routing tables) or just wait for it to be restarted by the Jail management tool (e.g. when using ``/usr/local/etc/rc.d/ezjail restart shell.foo.com`` as a reboot helper). + +---- + +Code +==== + +You can always get the latest version of *jaildaemon*, use ``git clone git://erdgeist.org/jaildaemon`` or the legacy view ``cvs -d :pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co jaildaemon`` with an empty password to check it out. There is a *jaildaemon* tarball, but no versioning yet. You can scroll through the source at `jaildaemon gitweb `_ or `jaildaemon cvsweb (deprecated) `_. Typing ``make install`` installs the binary in your ``${PREFIX}/bin`` (or ``/usr/local``, if none is set) and the rc-script to ``${PREFIX}/etc/rc.d/``. + +A tight integration with `ezjail `_ is, of course, following soon. + +---- + +Examples +======== + +Start the daemon as root with ``jaildaemon -D``, with optional parameters ``p`` and ``f`` to request the pidfile being written or to specify a different controlling command unix domain socket location (that is used by *jaildaemon* when run in client mode to communicate with the server, and defaults to ``/var/run/jaildaemon.pipe``), respectively. + +If you installed the package, you can just enable *jaildaemon* in your ``/etc/rc.conf``, by adding ``jaildaemon_enable=YES``. It should then start at reboot, or when you run ``/usr/local/etc/rc.d/jaildaemon start``. + +Start a new probe process in a Jail (in this example jid 23) to trigger the execution of the script:: + + /usr/local/bin/parse_route_config -s /usr/jails/shell.foo.com/ + +while identifying as 'route config update' in the jail and respawn:: + + jaildaemon -j 23 -c '/usr/local/bin/parse_route_config -s /usr/jails/shell.foo.com/' -t 'route config update' -r + +From within the jail (``ezjail-admin console shell.foo.com``) trigger the command by ``pkill -HUP -f 'route config update'``. + +Start a (non-spawning) Jail reboot trigger:: + + jaildaemon -j 23 -c '/usr/local/etc/rc.d/ezjail restart shell.foo.com' -t REBOOT + +Now, from within that Jail, reboot via ``pkill -HUP -f REBOOT``. + +---- + +Version history +=============== + +* Version 0.2: + + * Fixed a bug where *jaildaemon* when run from /etc/rc would ignore SIGHUP by default. + * You can now make the probe drop into another uid, so that non-root-processes can trigger actions, as well. + * Allow *jaildaemon* to accept the jailname for the -j parameter, as well. + +---- + +License +======= + +*jaildaemon* was written by `Dirk Engling `_ with lots of help from sg and is considered `beer ware `_. diff --git a/stories/projects/minimunin.rst b/stories/projects/minimunin.rst new file mode 100644 index 0000000..170c30a --- /dev/null +++ b/stories/projects/minimunin.rst @@ -0,0 +1,75 @@ +.. link: +.. description: is a tiny munin-node implementation for FreeBSD written in pure bourne shell providing basic plugin support. +.. tags: project +.. date: 2015/02/01 19:10:08 +.. title: minimunin +.. subtitle: A tiny munin-node implementation for FreeBSD written in pure bourne shell providing basic plugin support +.. slug: ../arts/software/minimunin/index +.. prio: 800 + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +Overview +======== + +.. image:: http://www.freebsd.org/gifs/daemon_hammer-tn25.jpg + :align: right + +`munin `_ is an open source multi platform service monitoring framework, written in perl. Munin nodes report server statistics to their respective masters. Unfortunally, the munin-node script is written in perl, too – even though the language features never actually are used. On a FreeBSD system this introduces unnecessary dependencies. *minimunin* tries to address this: It was implemented as a simple shell script triggered from inetd.conf and comes with some handy built in monitoring commands. Most of the munin plugins should also work. + +---- + +Code +==== + +You can always get the latest version of *minimunin*, use ``git clone git://erdgeist.org/minimunin` or the legacy view ``cvs -d :pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co minimunin`` with an empty password to check it out. There is a `minimunin tarball `_. You can scroll through the source at `minimunin gitweb ` or `minimunin cvsweb (deprecated) `_. Typing ``make install`` installs the binary in your ``${PREFIX}/bin`` (or ``/usr/local``, if none is set). + +---- + +Installation +============ + +After getting *minimunin* either from source or by installing the FreeBSD port/package, you need to activate *minimunin*. Just add the line:: + + munin stream tcp nowait nobody /usr/libexec/tcpd /usr/local/bin/minimunin + +to your ``/etc/inetd.conf``. Alternatively, if you plan to run plugins that require root permissions (e.g. for changing to a different user via the plugin config), add:: + + munin stream tcp nowait root /usr/libexec/tcpd /usr/local/bin/minimunin`` to your ``/etc/inetd.conf + +In both cases you need to enable inetd by setting ``inetd_enable="YES"`` in your ``/etc/rc.conf`` and restart inetd with the ``service inetd start`` command. You may also want to restrict access to your munin node via the ``/etc/hosts.allow`` and ``/etc/hosts.deny`` configuration files. + +---- + +Plugins +======= + +Munin has been designed to allow administrators to `write their own plugins `_ to provide statistics not covered by the base installation. You can find `many munin plugins here `_. *minimunin* can execute those plugins, as well. It expects plugin files in the folder /usr/local/etc/minimunin-plugins and will ignore `wild card plugins `_. In order to use them (after reading the documentation on the project website ;) create the appropriate soft link, e.g. ``ln -s if_ /usr/local/etc/minimunin-plugins/if_bge0``. Please note that the plugins must be executable and you may need to install further dependencies for plugins, such as perl, python or monitoring helpers. + +munin plugins `can be configured with ini-file style config files `_. *minimunin* expects config to reside under ``/usr/local/etc/minimunin-configs``, either as a single config file or a directory containing respective configs. Currently *minimunin* only supports the config parameters ``env.``-variables, ``user`` and ``command``. The ``group`` and ``timeout`` config parameters are not supported. The ``user`` parameter only works if *minimunin* finds itself being root and otherwise silently ignores the ``user`` parameter. + +---- + +Version history +=============== + +* V0.2 `minimunin-0.2.tar.bz2 `_ (2014-13-02) + + * Fixed a bug where uptime was not reported correctly. Thanks to Lothar Schmidt. + * Fixed a bug where only the last environment setting of a plugin config was obeyed. Thanks to Philipp Wünsche. +* V0.1 `minimunin-0.1.tar.bz2 `_ (2013-09-29) + + * First version, no diffs to previous ones ;) + +---- + +License +======= + +*minimunin* was written by `Dirk Engling `_ and is considered `beer ware `_. + +Please do not even think about hesitating to contact me via email or `@erdgeist `_ on twitter. diff --git a/stories/projects/opentracker.rst b/stories/projects/opentracker.rst new file mode 100644 index 0000000..294d2c9 --- /dev/null +++ b/stories/projects/opentracker.rst @@ -0,0 +1,136 @@ +.. link: +.. description: is a highly scalable tracker software for the bittorrent protocol, currently in use on the largest bulk trackers around. +.. tags: project +.. date: 2015/02/19 19:10:08 +.. title: opentracker +.. subtitle: An open and free bittorrent tracker +.. slug: ../arts/software/opentracker/index +.. prio: 1000 + +.. contents:: + :depth: 1 + :class: ezjail-toc + +---- + +.. image:: opentracker_beta3.png + :align: left + :width: 200px + +Overview +======== + +*opentracker* is a open and free `bittorrent tracker `_ project. It aims for minimal resource usage and is intended to run at your wlan router. Currently it is deployed as an open and free tracker instance. Read our `free and open tracker blog `_ and announce your torrents there (but do not hesitate to setup your own free trackers!). + +---- + +Version History +=============== + +*opentracker* is currently only available from CVS. A tar ball will be rolled soon. Use ``git clone git://erdgeist.org/opentracker`` or the legacy view ``cvs -d :pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co opentracker`` with an empty passwort to check it out. An `*opentracker* gitweb `_ and an `*opentracker* cvsweb (deprecated) `_ are available. + +---- + +Build instructions +================== + +Until *opentracker* is declared official release ready, the way to install it is:: + + cvs -d :pserver:cvs@cvs.fefe.de:/cvs -z9 co libowfat + cd libowfat + make + cd .. + # LEGACY: cvs -d:pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co opentracker + git clone git://erdgeist.org/opentracker + cd opentracker + make + +That should leave you with an exectuable called ``opentracker`` and one debug version ``opentracker.debug``. + +Some variables in *opentracker*'s Makefile control features and behaviour of *opentracker*. Here they are: + +* ``-DWANT_V6`` makes *opentracker* an IPv6-only tracker. More in the v6-section below. +* *opentracker* can deliver gzip compressed full scrapes. Enable this with ``-DWANT_COMPRESSION_GZIP`` option. +* Normally *opentracker* tracks any torrent announced to it. You can change that behaviour by enabling ONE of ``-DWANT_ACCESSLIST_BLACK`` or ``-DWANT_ACCESSLIST_WHITE``. Note, that you have to provide a whitelist file in order to make *opentracker* do anything in the latter case. More in the closed mode section below. +* *opentracker* can run in a cluster. Enable this behaviour by enabling ``-DWANT_SYNC_LIVE``. Note, that you have to configure your cluster before you can use *opentracker* when this option is on. +* Some statistics *opentracker* can provide are sensitive. You can restrict access to these statistics by enabling ``-DWANT_RESTRICT_STATS``. See section statistics for more details. +* Fullscrapes is bittorrent's way to query a tracker for all tracked torrents. Since it's in the standard, it is enabled by default. Disable it by commenting out ``-DWANT_FULLSCRAPE``. +* By default *opentracker* will only allow the connecting endpoint's IP address to be announced. Bittorrent standard allows clients to provide an IP address in its query string. You can make *opentracker* use this IP address by enabling ``-DWANT_IP_FROM_QUERY_STRING``. +* Some experimental or older, deprecated features can be enabled by the ``-DWANT_LOG_NETWORKS``, ``-DWANT_SYNC_SCRAPE`` or ``-DWANT_IP_FROM_PROXY`` switch. +* Currently there is some packages for some linux distributions and OpenBSD around, but some of them patch Makefile and default config to make *opentracker* closed by default. I explicitly don't endorse those packages and will not give support for problems stemming from these missconfigurations. + +---- + +Invocation +========== + +*opentracker* can be run by just typing ``./opentracker``. This will make *opentracker* bind to ``0.0.0.0:6969`` and happily serve all torrents presented to it. If ran as root, *opentracker* will immediately chroot to . (or any directory given with the -d option) and drop all priviliges after binding to whatever tcp or udp ports it is requested. + +When options were few, *opentracker* used to accept all of them from command line. While this still is possible for most options, using them is quite unhandy: an example invocation would look like ``./opentracker -i 23.23.23.7 -p 80 -P 80 -p 6969 -i 23.23.23.8 -p 80 -r http://www.mytorrentsite.com/ -d /usr/local/etc/opentracker -w mytorrents.list -A 127.0.0.1``. + +*opentracker* now uses a config file that you can provide with the ``-f`` switch. + +---- + +Config file +=========== + +*opentracker*'s config file is very straight forward and a very well documented example config can be found in the file `opentracker.conf.sample `_. + +---- + +Closed mode +=========== + +While personally I like my tracker to be open, I can see that there's people that want to control what torrents to track – or not to track. If you've compiled *opentracker* with one of the accesslist-options (see Build instructions above), you can control which torrents are tracked by providing a file that contains a list of human readable info_hashes. An example whitelist file would look like :: + + 0123456789abcdef0123456789abcdef01234567 + 890123456789abcdef0123456789abcdef012345 + +To make *opentracker* reload it's white/blacklist, send a ``SIGHUP`` unix signal. + +---- + +Statistics +========== + +Given its very network centric approach, talking to *opentracker* via http comes very naturally. Besides the ``/announce`` and ``/scrape`` paths, there is a third path you can access the tracker by: ``/stats``. This request takes parameters, for a quick overview just inquire `/stats?mode=everything``. + +Statistics have grown over time and are currently not very tidied up. Most modes were written to dump legacy-SNMP-style blocks that can easily be monitored by `MRTG `_. These modes are: ``peer``, ``conn``, ``scrp``, ``udp4``, ``tcp4``, ``busy``, ``torr``, ``fscr``, ``completed``, ``syncs``. I'm not going to explain these here. + +The ``statedump`` mode dumps non-recreatable states of the tracker so you can later reconstruct an *opentracker* session with the -l option. This is beta and wildly undocumented. + +You can inquire *opentracker*'s version (i.e. CVS versions of all its objects) using the ``version`` mode. + +---- + +Philosophy +========== + +A torrent tracker basically is an http-Server that collects all clients ip addresses into pools sorted by one of the request strings parameters and answers all other clients that specified this exact same parameter a list of all other recent clients. All technologies to implement this are around for more than twenty years. Still most implementations suck performancewise. + +Utilizing the highly scalable server framework from `libowfat `_, *opentracker* can easily serve multiple thousands of requests on a standard plastic WLAN-router, limited only by your kernels capabilities ;) + +One important design decision of *opentracker* was to not store any data persistently. This reduces wear&tear on hard disks and eliminates problems with corrupt databases. + +---- + +Author +====== + +*opentracker* was written by `Dirk Engling `_, who likes to hear from happy customers. + +---- + +Thanks +====== + +A project like this one is impossible without lots of help from friends. It is powered by beer, much energy and love, batches of bug reports and support on the operating system integration side by denis, taklamakan, cryx, supergrobi and – his libowfat always on the bleeding edge – Fefe. Thanks are also due to Hannes for helping me out with designing internal data representation. + +---- + +License +======= + +*opentracker* is considered `beer ware `_. + diff --git a/stories/projects/timestretch.rst b/stories/projects/timestretch.rst new file mode 100644 index 0000000..15ce5ca --- /dev/null +++ b/stories/projects/timestretch.rst @@ -0,0 +1,63 @@ +.. link: +.. description: is a fast implementation for SOLA, a sample time stretching algorithm. +.. tags: project +.. date: 2015/02/01 19:10:08 +.. title: timestretch +.. subtitle: a fast sample time stretching implementation +.. slug: ../arts/software/timestretch/index + +.. contents:: + :depth: 1 + :class: ezjail-toc + +Overview +======== + +When reducing or increasing the playback speed of an audio recording, it's pitch usually changes, leading to an audible "mickey mouse" effect. The `SOLA algorithm `_ provides a way to change the speed without altering the pitch. + +---- + +Details +======= + +The basic idea behind the method lies in the inherent redundancy in the periodic waves of spoken words or music. A vowel consists of overlayed and repeating waves where a human ear would not mind a single one of those repetitions being there or not: + +.. image:: timestretch_1.png + + +If we want to play a sample faster, we can try to find the period length corresponding to the base frequency of the recording in that place and overlap some periods, cross fading the first snippet one into the other. Or put it more visually intuitive: + +.. image:: timestretch_2.png + + +Obviously the recording is shorter than before, meaning it can be played in less time. In order to reduce the play time by a fixed ratio, we can alter the length of the overlapping windows (bright blue). + +To find the perfect offset to overlay our samples at, we simply brute force the sum of all differences of all samples for each offset, using the `mean squared error `_ and selecting the offset where the error is minimal – a process commonly called auto correlation. In our implementation this error is further biased towards the center of our window, so that the algorithm is not forced into selecting some sub optimal (and audibly bad) positions after a while, if the inherent period length the overlapping length differs only slightly. + +---- + +Build instructions +================== + +*timestretch* is available from my git repository. Use ``git clone git://erdgeist.org/timestretch`` to check it out. An `timestretch gitweb `_ is available. + +Currently there's only one source file containing the setup routine ``calc_convert_values``, which takes sample rate (used to calculate a proper window length in ms, based on heuristic values) and a tempo, which is a floating point value giving the rate in which to slow down or speed up the recording. It fills global variables you might want to put in a context struct for your project. At exit, the value ``g_input_length`` is the minimum number of samples expected for later processing, ``g_output_length`` is the exact amount of samples produced by each run. + +The actual work is done in the ``process_frame`` function which takes a pointer to the input and output buffer, the pointer to some scratch space (set up in ``calc_convert_values`` in this implementation) and a ``frame_flag`` indicating if this is the first frame (where nothing is there to cross fade with), a normal frame and the last frame, where no samples are kept for later cross fading and the caller can continue to resume to non-timestretched playback again. + +---- + +Author +====== + +*timestretch* was written by `Dirk Engling `_, who likes to hear from happy customers. + +---- + +License +======= + +*timestretch* is considered `beer ware `_. + + + diff --git a/stories/projects/vchat-client.rst b/stories/projects/vchat-client.rst new file mode 100644 index 0000000..f540f04 --- /dev/null +++ b/stories/projects/vchat-client.rst @@ -0,0 +1,267 @@ +.. link: +.. description: is a curses based chat client for the arcane vchat protocol. +.. tags: project +.. date: 2014/02/01 19:10:08 +.. title: vchat-client +.. slug: ../arts/software/vchat-client/index + +Newbies Guide to vchat via 'Buntclient' +======================================= + +1. Get the Source, Luke +----------------------- + +Make sure, you got a shell and the tool called cvs. +cd to a directory the source shall reside in. + +Type:: + + $ cvs -d:pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot login + $ cvs -z3 -d:pserver:anoncvs@cvs.erdgeist.org:/home/cvsroot co vchat-client + +(press return at password prompt) + + +2. Do the verpile +----------------- + +Enter the vchat-client/ directory. +Type:: + + $ make + +If that fails due to some readline fuckups, try again with:: + + $ make OLDREADLINE=-DOLDREADLINE + +If you are administrator of your computer and want to install +vchat-client permanently, type:: + + $ make install + +You may also run the binary from its compile location. + + +3. Get protected +---------------- + +Run the script vchat-keygen, located in the vchat-client/ directory. +Follow its instructions. + + +4. Home sweet Home +------------------ + +Create a .vchat/ directory in your homedir. :: + + $ cd + $ mkdir .vchat/ + +Copy your key to the .vchat/ directory:: + + $ cp ~/.vchat.key ~/.vchat/key + +Whenever your signed cert arrives from vchat at vchat.berlin.ccc dot de, +copy it to the .vchat/ directory as well:: + + $ cp ~/.vchat.cert ~/.vchat/cert + +In order to modify the look of your client, you may copy one of +the sample-xxx.fmt files to the .vchat/ directory:: + + $ cd vchat-client/ + $ cp sample-erdgeist.fmt ~/.vchat/formats + +If you want to set the chat server to vchat.berlin.ccc.de now, +type:: + + $ echo host=vchat.berlin.ccc.de >> ~/.vchat/config + +If you want to ignore SSL-warnings due to missing CA-files, type:: + + $ echo ignssl=1 >> ~/.vchat/config + +If you don't want to ignore SSL-warnings, get the root-certificates from: + + http://www.cacert.org/certs/class3.txt + and + http://www.cacert.org/certs/root.txt + +and copy them into your openssl-certs directory. For example:: + + # cp root.txt /etc/ssl/certs/ + # cp class3.txt /etc/ssl/certs/ + # cd /etc/ssl/certs + # ln -s root.txt `openssl x509 -in root.txt -hash | head -n 1`.0 + # ln -s class3.txt `openssl x509 -in class3.txt -hash | head -n 1`.0 + +Now you can type:: + + $ echo ignssl=0 >> ~/.vchat/config + + +If you want a seperate private message window, type:: + + $ echo messages=10 >> ~/.vchat/config + +If you want to have no topic line, like in the oldschool client, +do not choose a private message window an type:: + + $ echo usetopicbar=0 >> ~/.vchat/config + +5. First bunt +------------- + +Check, whether you REALLY got a signed cert. + +[.... +If you did not get a reply from vchat at vchat.berlin.ccc dot de within +3 days, this might either be due to you not being known to the +certificate masters, or simply because of the verpeil. You may, +after 3 days, contact me at erdgeist at erdgeist dot org. If I don't +reply, too, reconsider your lifestyle. +....] + +Start the client:: + + $ vchat-client + +or :: + + $ vchat-client/vchat-client + +Enter the passphrase to your private key, if you added one. + +Look around. If the Umlautz are broken, try to trouble shoot here: +https://freepad.erdgeist.org/p/fT9OoDoLMx + +Try to feel comfortable with the new look of your vchat- +environment. Close your mouth. Try playing around with the +following commands: + +If you did not enable the seperate private message window, +ignore this part: + +[ STARTIGNORE + +Press Ctrl-G and Ctrl-T several times. +Press Ctrl-X, then send yourself a private message, watch, what +happens +Press Ctrl-C ONCE!!! +Press Ctrl-R and watch the asteriks at the very right of your +screen, this one indicates, which window to scroll back + +ENDIGNORE] + +Press Ctrl-B Ctrl-F, if you collected some text in your +message window. This should scroll back and forth. +A red console bar should indicate, that you scrolled up. + +Press Ctrl-L to redraw and reset scroll indices. + +Play around with the completion: + +Hitting at start of line will complete to :: + + .m + +if you hit more often, you will cycle through this list, +sorted by the last time you exchanges PMs with. + +Hitting when you already typed some letters but no space +yet will expand to all nicks in your current channel. You may +cycle through this list, too. + +Hitting tab anywhere else expands to all nicks in all channels. + +Most other readline hotkeys (Ctrl-U, Ctrl-K, Ctrl-W, Ctrl-A...) +will work as expected. + + +6. Commandoe +------------ + +In addition to the server commands (.h, .m, .a, .s ...) the +client brings its own set of commands, which will be triggered +IRC-style by /COMMAND. These may be abbrevated to the lowest +unambigous substring. + +Try /HELP for a first glimpse. +The status window may be forced off by hitting Ctrl-X. + +/HELP KEYS should give you a short summary of available hot +key commands. + +Some client commands are wrappers around server commands:: + + /ME == .a + /MSG == .m + /M == .m + /QUIT == .x + +The other commands are used to enable client side filtering +and highlighting. + +Try:: + + /HELP FILTERS + /HELP FLT + /HELP FILTERS + +Highlight yourself:: + + /FLT 6 + +if you don't see anything getting highlighted, you probably +have a mac. If you got this far despite of that problem, you +will surely find a way to enable colors in your terminal. + +Remove the filter rule:: + + /rmflt 1 + +Zoom in all channel leaves:: + + /flt + left channel + +List your filter rules:: + + /lsflt + +Remove all filters:: + + /clflt + +Ignore all server leave and join messages:: + + /flt - ((left|joined|entered) (the )?(channel|chat)) + +Reenable those lines:: + + /rmflt ((left|joined|entered) (the )?(channel|chat)) + +Note: zoom overrides ignore. + +Be careful about setting your filters, as these are treated +as regular expressions, which easily may lead to time +consuming evaluations. Avoid brackets. + + +7. Fiiiiiiiiieschas +------------------- + +As I am still young *cough cough* I am _STILL_ willing to +implement features of all stupidity grades. Contact me +with /m erdgeist at any time. + + +8. You help +----------- + +Since you will probably start fiddeling around with your +.vchat/formats file, I'd appreciate getting YOUR format, +too, as this may be helpful for others to make their own. +Just send them to erdgeist at erdgeist dot org. Bug reports, +feature request that are to long for the chat and diffs +implementing some features are welcome at this address, +too. -- cgit v1.2.3