Merge commit 'tor-0.2.2.1-alpha' into debian-merge

* commit 'tor-0.2.2.1-alpha': (187 commits) HiddenServiceVersion must be set to 2 currently. put karsten's changelog in the right place When Tor fails to parse a descriptor of any kind, dump it to disk. update fetch-all with dir auth changelog and spec changes for the .exit fix bump to 0.2.2.1-alpha typos in dir-spec mark off a done proposal clean up the changelog for 0.2.2.1-alpha A changelog entry and a bit more documentation for socks-client Clean up a couple of style issues in the socks-client branch. new proposals: params in consensus, and lower circwindow Add some fixes after discussion with Nick. Refactor geoip_get_dirreq_history() some more. Fix a memory leak in summarizing directory request timing. Add the first 8 bytes of the git commit digest to our versions. autoconf 2.59 appears not to support AC_PROG_SED Survive unparseable cached cert file Notice v3 cert parsing failures Make crypto_digest_get_digest nondestructive again. ...
2024-11-20 10:12:15 +01:00 · 2009-09-03 14:45:54 +02:00 · 2009-09-03 14:45:54 +02:00 · 1407022162
commit 1407022162
parent 58ccdfccef 1092fdca53
175 changed files with 7176 additions and 2432 deletions
--- a/.gitignore
+++ b/.gitignore
@ -108,8 +108,10 @@
 # /src/common/
 /src/common/Makefile
 /src/common/Makefile.in
+/src/common/common_sha1.i
 /src/common/libor.a
 /src/common/libor-crypto.a
+/src/common/libor-event.a

 # /src/config/
 /src/config/Makefile
@ -121,6 +123,7 @@
 # /src/or/
 /src/or/Makefile
 /src/or/Makefile.in
+/src/or/or_sha1.i
 /src/or/micro-revision.*
 /src/or/tor
 /src/or/test
--- a/125
+++ b/125
@ -1,7 +1,120 @@
-Changes in version 0.2.1.19 - 2009-07-28
+Changes in version 0.2.2.1-alpha - 2009-08-26
+  o Security fixes:
+    - Start the process of disabling ".exit" address notation, since it
+      can be used for a variety of esoteric application-level attacks
+      on users. To reenable it, set "AllowDotExit 1" in your torrc. Fix
+      on 0.0.9rc5.
+
+  o New directory authorities:
+    - Set up urras (run by Jacob Appelbaum) as the seventh v3 directory
+      authority.
+
+  o Major features:
+    - New AccelName and AccelDir options add support for dynamic OpenSSL
+      hardware crypto acceleration engines.
+    - Tor now supports tunneling all of its outgoing connections over
+      a SOCKS proxy, using the SOCKS4Proxy and/or SOCKS5Proxy
+      configuration options. Code by Christopher Davis.
+
  o Major bugfixes:
-    - Make accessing hidden services on 0.2.1.x work right
-      again. Bugfix on 0.2.1.3-alpha; workaround for bug 1038.
+    - Send circuit or stream sendme cells when our window has decreased
+      by 100 cells, not when it has decreased by 101 cells. Bug uncovered
+      by Karsten when testing the "reduce circuit window" performance
+      patch. Bugfix on the 54th commit on Tor -- from July 2002,
+      before the release of Tor 0.0.0. This is the new winner of the
+      oldest-bug prize.
+
+  o New options for gathering stats safely:
+    - Directories that set "DirReqStatistics 1" write statistics on
+      directory request to disk every 24 hours. As compared to the
+      --enable-geoip-stats flag in 0.2.1.x, there are a few improvements:
+      1) stats are written to disk exactly every 24 hours; 2) estimated
+      shares of v2 and v3 requests are determined as mean values, not at
+      the end of a measurement period; 3) unresolved requests are listed
+      with country code '??'; 4) directories also measure download times.
+    - Exit nodes that set "ExitPortStatistics 1" write statistics on the
+      number of exit streams and transferred bytes per port to disk every
+      24 hours.
+    - Relays that set "CellStatistics 1" write statistics on how long
+      cells spend in their circuit queues to disk every 24 hours.
+    - Entry nodes that set "EntryStatistics 1" write statistics on the
+      rough number and origins of connecting clients to disk every 24
+      hours.
+    - Relays that write any of the above statistics to disk and set
+      "ExtraInfoStatistics 1" include the past 24 hours of statistics in
+      their extra-info documents.
+
+  o Minor features:
+    - New --digests command-line switch to output the digests of the
+      source files Tor was built with.
+    - The "torify" script now uses torsocks where available.
+    - The memarea code now uses a sentinel value at the end of each area
+      to make sure nothing writes beyond the end of an area. This might
+      help debug some conceivable causes of bug 930.
+    - Time and memory units in the configuration file can now be set to
+      fractional units. For example, "2.5 GB" is now a valid value for
+      AccountingMax.
+    - Certain Tor clients (such as those behind check.torproject.org) may
+      want to fetch the consensus in an extra early manner. To enable this
+      a user may now set FetchDirInfoExtraEarly to 1. This also depends on
+      setting FetchDirInfoEarly to 1. Previous behavior will stay the same
+      as only certain clients who must have this information sooner should
+      set this option.
+    - Instead of adding the svn revision to the Tor version string, report
+      the git commit (when we're building from a git checkout).
+
+  o Minor bugfixes:
+    - If any the v3 certs we download are unparseable, we should actually
+      notice the failure so we don't retry indefinitely. Bugfix on
+      0.2.0.x; reported by "rotator".
+    - If the cached cert file is unparseable, warn but don't exit.
+    - Fix possible segmentation fault on directory authorities. Bugfix on
+      0.2.1.14-rc.
+    - When Tor fails to parse a descriptor of any kind, dump it to disk.
+      Might help diagnosing bug 1051.
+
+  o Deprecated and removed features:
+    - The controller no longer accepts the old obsolete "addr-mappings/"
+      or "unregistered-servers-" GETINFO values.
+    - Hidden services no longer publish version 0 descriptors, and clients
+      do not request or use version 0 descriptors. However, the old hidden
+      service authorities still accept and serve version 0 descriptors
+      when contacted by older hidden services/clients.
+    - The EXTENDED_EVENTS and VERBOSE_NAMES controller features are now
+      always on; using them is necessary for correct forward-compatible
+      controllers.
+    - Remove support for .noconnect style addresses. Nobody was using
+      them, and they provided another avenue for detecting Tor users
+      via application-level web tricks.
+
+
+Changes in version 0.2.1.20 - 2009-??-??
+  o Major bugfixes:
+    - Send circuit or stream sendme cells when our window has decreased
+      by 100 cells, not when it has decreased by 101 cells. Bug uncovered
+      by Karsten when testing the "reduce circuit window" performance
+      patch. Bugfix on the 54th commit on Tor -- from July 2002,
+      before the release of Tor 0.0.0. This is the new winner of the
+      oldest-bug prize.
+
+  o New directory authorities:
+    - Set up urras (run by Jacob Appelbaum) as the seventh v3 directory
+      authority.
+
+  o Minor bugfixes:
+    - Fix a signed/unsigned compile warning in 0.2.1.19.
+    - Fix possible segmentation fault on directory authorities. Bugfix on
+      0.2.1.14-rc.
+
+
+Changes in version 0.2.1.19 - 2009-07-28
+  Tor 0.2.1.19 fixes a major bug with accessing and providing hidden
+  services on Tor 0.2.1.3-alpha through 0.2.1.18.
+
+  o Major bugfixes:
+    - Make accessing hidden services on 0.2.1.x work right again.
+      Bugfix on 0.2.1.3-alpha; workaround for bug 1038. Diagnosis and
+      part of patch provided by "optimist".

  o Minor features:
    - When a relay/bridge is writing out its identity key fingerprint to
@ -20,6 +133,12 @@ Changes in version 0.2.1.19 - 2009-07-28


 Changes in version 0.2.1.18 - 2009-07-24
+  Tor 0.2.1.18 lays the foundations for performance improvements,
+  adds status events to help users diagnose bootstrap problems, adds
+  optional authentication/authorization for hidden services, fixes a
+  variety of potential anonymity problems, and includes a huge pile of
+  other features and bug fixes.
+
  o Build fixes:
    - Add LIBS=-lrt to Makefile.am so the Tor RPMs use a static libevent.

--- a/Doxyfile.in
+++ b/Doxyfile.in
@ -1,4 +1,3 @@
-# $Id$
 # Doxyfile 1.5.1

 # This file describes the settings to be used by the documentation system
--- a/Makefile.am
+++ b/Makefile.am
@ -1,4 +1,3 @@
-# $Id$
 # Copyright (c) 2001-2004, Roger Dingledine
 # Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson
 # Copyright (c) 2007-2009, The Tor Project, Inc.
--- a/31
+++ b/31
@ -3,7 +3,38 @@ This document summarizes new features and bugfixes in each stable release
 of Tor. If you want to see more detailed descriptions of the changes in
 each development snapshot, see the ChangeLog file.

+Changes in version 0.2.1.19 - 2009-07-28
+  Tor 0.2.1.19 fixes a major bug with accessing and providing hidden
+  services.
+
+  o Major bugfixes:
+    - Make accessing hidden services on 0.2.1.x work right again.
+      Bugfix on 0.2.1.3-alpha; workaround for bug 1038. Diagnosis and
+      part of patch provided by "optimist".
+
+  o Minor features:
+    - When a relay/bridge is writing out its identity key fingerprint to
+      the "fingerprint" file and to its logs, write it without spaces. Now
+      it will look like the fingerprints in our bridges documentation,
+      and confuse fewer users.
+
+  o Minor bugfixes:
+    - Relays no longer publish a new server descriptor if they change
+      their MaxAdvertisedBandwidth config option but it doesn't end up
+      changing their advertised bandwidth numbers. Bugfix on 0.2.0.28-rc;
+      fixes bug 1026. Patch from Sebastian.
+    - Avoid leaking memory every time we get a create cell but we have
+      so many already queued that we refuse it. Bugfix on 0.2.0.19-alpha;
+      fixes bug 1034. Reported by BarkerJr.
+
+
 Changes in version 0.2.1.18 - 2009-07-24
+  Tor 0.2.1.18 lays the foundations for performance improvements,
+  adds status events to help users diagnose bootstrap problems, adds
+  optional authentication/authorization for hidden services, fixes a
+  variety of potential anonymity problems, and includes a huge pile of
+  other features and bug fixes.
+
  o Major features (clients):
    - Start sending "bootstrap phase" status events to the controller,
      so it can keep the user informed of progress fetching directory
--- a/acinclude.m4
+++ b/acinclude.m4
@ -1,4 +1,3 @@
-dnl $Id$
 dnl Helper macros for Tor configure.in
 dnl Copyright (c) 2001-2004, Roger Dingledine
 dnl Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson
--- a/configure.in
+++ b/configure.in
@ -1,11 +1,10 @@
-dnl $Id$
 dnl Copyright (c) 2001-2004, Roger Dingledine
 dnl Copyright (c) 2004-2006, Roger Dingledine, Nick Mathewson
 dnl Copyright (c) 2007-2008, The Tor Project, Inc.
 dnl See LICENSE for licensing information

 AC_INIT
-AM_INIT_AUTOMAKE(tor, 0.2.1.19)
+AM_INIT_AUTOMAKE(tor, 0.2.2.1-alpha)
 AM_CONFIG_HEADER(orconfig.h)

 AC_CANONICAL_HOST
@ -86,13 +85,6 @@ case $host in
     ;;
 esac

-AC_ARG_ENABLE(geoip-stats,
-     AS_HELP_STRING(--enable-geoip-stats, enable code for directories to collect per-country statistics))
-
-if test "$enable_geoip_stats" = "yes"; then
-  AC_DEFINE(ENABLE_GEOIP_STATS, 1, [Defined if we try to collect per-country statistics])
-fi
-
 AC_ARG_ENABLE(gcc-warnings,
     AS_HELP_STRING(--enable-gcc-warnings, enable verbose warnings))

@ -108,6 +100,12 @@ AC_PROG_CPP
 AC_PROG_MAKE_SET
 AC_PROG_RANLIB

+dnl autoconf 2.59 appears not to support AC_PROG_SED
+AC_CHECK_PROG([SED],[sed],[sed],[/bin/false])
+
+AC_PATH_PROG([SHA1SUM], [sha1sum], none)
+AC_PATH_PROG([OPENSSL], [openssl], none)
+
 TORUSER=_tor
 AC_ARG_WITH(tor-user,
        [  --with-tor-user=NAME    Specify username for tor daemon ],
@ -270,15 +268,19 @@ save_CPPFLAGS="$CPPFLAGS"
 LIBS="-levent $TOR_LIB_WS32 $LIBS"
 LDFLAGS="$TOR_LDFLAGS_libevent $LDFLAGS"
 CPPFLAGS="$TOR_CPPFLAGS_libevent $CPPFLAGS"
-AC_CHECK_FUNCS(event_get_version event_get_method event_set_log_callback)
+AC_CHECK_FUNCS(event_get_version event_get_version_number event_get_method event_set_log_callback evdns_set_outgoing_bind_address event_base_loopexit)
 AC_CHECK_MEMBERS([struct event.min_heap_idx], , ,
 [#include <event.h>
 ])

+AC_CHECK_HEADERS(event2/event.h event2/dns.h)
+
 LIBS="$save_LIBS"
 LDFLAGS="$save_LDFLAGS"
 CPPFLAGS="$save_CPPFLAGS"

+AM_CONDITIONAL(USE_EXTERNAL_EVDNS, test x$ac_cv_header_event2_dns_h = xyes)
+
 dnl ------------------------------------------------------
 dnl Where do you live, openssl?  And how do we call you?

--- a/contrib/checkOptionDocs.pl
+++ b/contrib/checkOptionDocs.pl
@ -1,5 +1,4 @@
 #!/usr/bin/perl -w
-# $Id
 use strict;

 my %options = ();
--- a/contrib/checkSpace.pl
+++ b/contrib/checkSpace.pl
@ -42,9 +42,8 @@ for $fn (@ARGV) {
            $lastnil = 0;
        }
        ## Terminals are still 80 columns wide in my world.  I refuse to
-        ## accept double-line lines.  Except, of course, svn Id tags
-        ## can make us go long.
-        if (/^.{80}/ && !/\$Id: /) {
+        ## accept double-line lines.
+        if (/^.{80}/) {
            print "     Wide:$fn:$.\n";
        }
        ### Juju to skip over comments and strings, since the tests
--- a/contrib/cross.sh
+++ b/contrib/cross.sh
@ -1,5 +1,4 @@
 #!/bin/bash
-# $Id$
 # Copyright 2006 Michael Mohr with modifications by Roger Dingledine
 # See LICENSE for licensing information.

--- a/contrib/directory-archive/fetch-all
+++ b/contrib/directory-archive/fetch-all
@ -30,8 +30,8 @@ DIRSERVERS=""
 DIRSERVERS="$DIRSERVERS 86.59.21.38:80"		# tor26
 DIRSERVERS="$DIRSERVERS 128.31.0.34:9031"	# moria1
 DIRSERVERS="$DIRSERVERS 128.31.0.34:9032"	# moria2
-#DIRSERVERS="$DIRSERVERS 140.247.60.64:80"	# lefkada
 DIRSERVERS="$DIRSERVERS 194.109.206.212:80"	# dizum
+
 DATEDIR=$(date "+%Y/%m/%d")
 TIME=$(date "+%Y%m%d-%H%M%S")

--- a/contrib/directory-archive/fetch-all-v3
+++ b/contrib/directory-archive/fetch-all-v3
@ -35,6 +35,7 @@ DIRSERVERS="$DIRSERVERS 80.190.246.100:80"	# gabelmoo
 DIRSERVERS="$DIRSERVERS 194.109.206.212:80"	# dizum
 #DIRSERVERS="$DIRSERVERS 128.31.0.34:9032"	# moria2
 DIRSERVERS="$DIRSERVERS 213.73.91.31:80"	# dannenberg
+DIRSERVERS="$DIRSERVERS 208.83.223.34:443"	# urras
 TIME=$(date "+%Y%m%d-%H%M%S")

 . fetch-all-functions
--- a/contrib/id_to_fp.c
+++ b/contrib/id_to_fp.c
@ -1,5 +1,4 @@
 /* Copyright 2006 Nick Mathewson; see LICENSE for licensing information */
-/* $Id$ */

 /* id_to_fp.c : Helper for directory authority ops.  When somebody sends us
 * a private key, this utility converts the private key into a fingerprint
--- a/contrib/nagios-check-tor-authority-cert
+++ b/contrib/nagios-check-tor-authority-cert
@ -8,8 +8,6 @@
 # Usage: nagios-check-tor-authority-cert <authority identity fingerprint>
 # e.g.: nagios-check-tor-authority-cert A9AC67E64B200BBF2FA26DF194AC0469E2A948C6

-# $Id$
-
 # Copyright (c) 2008 Peter Palfrader <peter@palfrader.org>
 #
 # Permission is hereby granted, free of charge, to any person obtaining
--- a/contrib/osx/package.sh
+++ b/contrib/osx/package.sh
@ -1,5 +1,4 @@
 #!/bin/sh
-# $Id$
 # Copyright 2004-2005 Nick Mathewson. 
 # Copyright 2005-2007 Andrew Lewman
 # Copyright 2008 The Tor Project, Inc.
--- a/contrib/osx/uninstall_tor_bundle.sh
+++ b/contrib/osx/uninstall_tor_bundle.sh
@ -134,7 +134,7 @@ fi
 ## clean up
 echo ". Cleaning up"
 rm -rf $TEMP_BOM_CONTENTS
-rm -rf /Library/Privoxy/ /Library/StartupItems/Privoxy/ /Library/Tor/ /Library/StartupItems/Tor/ /Library/Torbutton/ 
+rm -rf /Library/Privoxy/ /Library/StartupItems/Privoxy/ /Library/Tor/ /Library/StartupItems/Tor/ /Library/Torbutton/ /Library/Receipts/Privoxy.pkg /Library/Receipts/torbutton.pkg /Library/Receipts/Tor.pkg /Library/Receipts/Vidalia.pkg /Library/Receipts/TorStartup.pkg

 echo ". Finished"

--- a/contrib/polipo/package.sh
+++ b/contrib/polipo/package.sh
@ -1,5 +1,4 @@
 #!/bin/sh
-# $Id: package.sh 8992 2006-12-23 03:12:09Z phobos $
 # Copyright 2004-2005 Nick Mathewson & Andrew Lewman. 
 # Copyright 2005-2008 Andrew Lewman
 # This is licensed under the Modified BSD License.
--- a/contrib/rc.subr
+++ b/contrib/rc.subr
@ -1,5 +1,4 @@
 #!/bin/sh
-# $Id$
 # $FreeBSD: ports/security/tor-devel/files/tor.in,v 1.1 2006/02/17 22:21:25 mnag Exp $
 #
 # (rc.subr written by Peter Thoenen for Net/FreeBSD)
--- a/contrib/tor-mingw.nsi.in
+++ b/contrib/tor-mingw.nsi.in
@ -9,7 +9,7 @@
 !include "FileFunc.nsh"
 !insertmacro GetParameters
  
-!define VERSION "0.2.1.19"
+!define VERSION "0.2.2.1-alpha"
 !define INSTALLER "tor-${VERSION}-win32.exe"
 !define WEBSITE "https://www.torproject.org/"
 !define LICENSE "LICENSE"
@ -216,6 +216,7 @@ Function un.InstallFiles
 	Delete "$INSTDIR\tor.ico"
 	Delete "$SMSTARTUP\Tor.lnk"
 	Delete "$INSTDIR\Uninstall.exe"
+    Delete "$INSTDIR\geoip"
 FunctionEnd

 Function un.InstallDirectories
--- a/contrib/tor-resolve.py
+++ b/contrib/tor-resolve.py
@ -1,5 +1,4 @@
 #!/usr/bin/python
-#$Id$

 import socket
 import struct
--- a/contrib/torify.1
+++ b/contrib/torify.1
@ -1,22 +1,27 @@
 .TH torify 1 "" Jan-2009 ""
-.\" manual page by Peter Palfrader
+.\" manual page by Peter Palfrader and Jacob Appelbaum
 .SH NAME
 .LP
-torify \- wrapper for tsocks and tor
+torify \- wrapper for torsocks or tsocks and tor

 .SH SYNOPSIS
 \fBtorify\fP\ \fIapplication\fP\ [\fIapplication's\ arguments\fP]

 .SH DESCRIPTION
-\fBtorify\fR is a simple wrapper that calls tsocks with a tor specific
+\fBtorify\fR is a simple wrapper that attempts to find the best underlying Tor
+wrapper available on a system. It calls torsocks or tsocks with a tor specific
 configuration file.

+torsocks is an improved wrapper that explictly rejects UDP, safely resolves DNS
+lookups and properly socksifies your TCP connections.
+
 tsocks itself is a wrapper between the tsocks library and the application
 that you would like to run socksified.

-Please note that since tsocks uses LD_PRELOAD, torify cannot be applied
+Please note that since both method use LD_PRELOAD, torify cannot be applied
 to suid binaries.

+.SH WARNING
 You should also be aware that the way tsocks currently works only TCP
 connections are socksified.  Be aware that this will in most circumstances
 not include hostname lookups which would still be routed through your
@ -25,8 +30,13 @@ normal system resolver to your usual resolving nameservers.  The
 The Tor FAQ at https://wiki.torproject.org/noreply/TheOnionRouter/TorFAQ might
 have further information on this subject.

+When used with torsocks, torify should not leak DNS requests or UDP data.
+
+Both will leak ICMP data.
+
 .SH SEE ALSO
 .BR tor (1),
 .BR tor-resolve (1),
+.BR torsocks (1),
 .BR tsocks (1),
 .BR tsocks.conf (5).
--- a/contrib/torify.in
+++ b/contrib/torify.in
@ -10,36 +10,67 @@

 # Define and ensure we have tsocks
 # XXX: what if we don't have which?
+TORSOCKS="`which torsocks`"
 TSOCKS="`which tsocks`"
+PROG=""
 if [ ! -x "$TSOCKS" ]
 then
-        echo "$0: Can't find tsocks in PATH. Perhaps you haven't installed it?" >&2
-        exit 1
+	echo "$0: Can't find tsocks in PATH. Perhaps you haven't installed it?" >&2
+else
+	PROG=$TSOCKS
+fi
+if [ ! -x "$TORSOCKS" ]
+then
+	echo "$0: Can't find torsocks in PATH. Perhaps you haven't installed it?" >&2
+else
+	PROG=$TORSOCKS
+fi
+
+if [ ! -x "$PROG" ]
+then
+	echo "$0: Can't find the required tor helpers in our PATH. Perhaps you haven't installed them?" >&2
+	exit 1;
 fi

 # Check for any argument list
 if [ "$#" = 0 ]
 then
-        echo "Usage: $0 <command> [<options>...]" >&2
-        exit 1
+	echo "Usage: $0 [-hv] <command> [<options>...]" >&2
+	exit 1
 fi
 if [ "$#" = 1 ] && ( [ "$1" = "-h" ] || [ "$1" = "--help" ] )
 then
-        echo "Usage: $0 <command> [<options>...]"
-        exit 0
+	echo "Usage: $0 [-hv] <command> [<options>...]"
+	exit 0
 fi

-# Define our tsocks config file
-TSOCKS_CONF_FILE="@CONFDIR@/tor-tsocks.conf"
-export TSOCKS_CONF_FILE
-
-# Check that we've got a tsocks config file
-if [ -r "$TSOCKS_CONF_FILE" ]
+if [ "$1" = "-v" ] || [ "$1" = "--verbose" ]
 then
-	exec tsocks "$@"
-	echo "$0: Failed to exec tsocks $@" >&2
-	exit 1
-else
-	echo "$0: Missing tsocks configuration file \"$TSOCKS_CONF_FILE\"." >&2
-	exit 1
+	echo "We're armed with the following tsocks: $TSOCKS"
+	echo "We're armed with the following torsocks: $TORSOCKS"
+	echo "We're attempting to use $PROG for all tor action."
+	shift 1
+fi
+
+if [ "$PROG" = "$TSOCKS" ]
+then
+	# Define our tsocks config file
+	TSOCKS_CONF_FILE="/etc/tor/tor-tsocks.conf"
+	export TSOCKS_CONF_FILE
+
+	# Check that we've got a tsocks config file
+	if [ -r "$TSOCKS_CONF_FILE" ]
+	then
+		echo "WARNING: tsocks is known to leak DNS and UDP data." >&2
+		exec tsocks "$@"
+		echo "$0: Failed to exec tsocks $@" >&2
+		exit 1
+	else
+		echo "$0: Missing tsocks configuration file \"$TSOCKS_CONF_FILE\"." >&2
+		exit 1
+	fi
+fi
+if [ "$PROG" = "$TORSOCKS" ]
+then
+	exec torsocks "$@"
 fi
--- a/doc/HACKING
+++ b/doc/HACKING
@ -11,12 +11,20 @@
 0.1. Useful command-lines that are non-trivial to reproduce but can
 help with tracking bugs or leaks.

+0.1.1. Dmalloc
+
 dmalloc -l ~/dmalloc.log
 (run the commands it tells you)
 ./configure --with-dmalloc

+0.2.2. Valgrind
+
 valgrind --leak-check=yes --error-limit=no --show-reachable=yes src/or/tor

+(Note that if you get a zillion openssl warnings, you will also need to
+ pass --undef-value-errors=no to valgrind, or rebuild your openssl
+ with -DPURIFY.)
+
 0.2. Running gcov for unit test coverage

  make clean
--- a/doc/TODO
+++ b/doc/TODO
@ -4,8 +4,8 @@ We've split out our TODO into three files:
 TODO.02x is the list of items we're planning to get done in the next
 stable release.

-TODO.external is the list of external constraints and deliverables that
-we all need to keep in mind.
+TODO.external lives in svn under /projects/todo/. It's the list of
+external constraints and deliverables that we all need to keep in mind.

 TODO.future is the list of other items we plan to get to in later releases.

--- a/doc/TODO.021
+++ b/doc/TODO.021
@ -1,4 +1,3 @@
-$Id$
 Legend:
 SPEC!!  - Not specified
 SPEC    - Spec not finalized
--- a/doc/TODO.022
+++ b/doc/TODO.022
@ -8,14 +8,17 @@ NOTE 2: It's easy to list stuff like this with no time estimates and
        0.2.2, figure out how long the stuff we want will take, and
        triage accordingly, or vice versa.

- Design
+- Design only
  - Begin design work for UDP transition; identify areas where we need to
    make changes or instrument stuff early.
+    [multiple weeks, ongoing.  Need to do a draft early.]

 - Performance, mostly protocol-neutral.
  - Work with Libevent 2.0's bufferevent interface
    - Identify any performance stuff we need to push back into
      libevent to make it as fast as we want.
+    - Get a decent rate-limiting feature into Libevent
+    - Get openssl support into Libevent.

  - Revise how we do bandwidth limiting and round-robining between
    circuits on a connection.
@ -30,21 +33,76 @@ NOTE 2: It's easy to list stuff like this with no time estimates and

  - Figure out good ways to instrument Tor internals so we can tell
    how well our bandwidth and flow-control stuff is actually working.
+    - What ports eat the bandwidth?
+    - How full do queues get?
+    - How much latency do queues get?

- Features
+  - Rate limit at clients:
+    - Give clients an upper bound on how much they're willing to use
+      the network if they're not relaying?
+    - ... or group client circuits by IP at the server and rate-limit
+      like that.
+
+  - Use if-modified-since to download consensuses
+
+
+- Other features
  - Proposals to implement:
-    - 146: reflect long-term stability
+    - 146: reflect long-term stability in consensuses
    - 147: Stop using v2 directories to generate v3 votes.
+      - Start pinging as soon as we learn about a relay, not on a
+        22-minute cycle.  Prioritize new and volatile relays for
+        testing.

  - Proposals to improve and implement
    - 158: microdescriptors
+      o Revise proposal
+      - Implement
+    o 160: list bandwidth in consensus
+      o Finish proposal
+      o and actually set it reasonably
+      o and actually use it.

  - Proposals to improve and implement if not broken
-    - IPv6 support.  (Parts of 117, but figure out how to handle DNS
+    D IPv6 support.  (Parts of 117, but figure out how to handle DNS
      requests.)
    - 140: Directory diffs
+      - Need a decent simple C diff implementation.
+      - Need a decent simple C ed patch implementation.
    - 149: learn info from netinfo cells.
-    - 134: handle authority fragmentation (Needs more analysis)
+      o Start discussion
+      - Revise proposal based on discussion.
+    X 134: handle authority fragmentation (Needs more analysis)
+    - 165: Easy migration for voting authority sets
+    - 163: Detect client-status better
+      o Write proposal
+      - Possibly implement, depending on discussion.
+    - 164: Have authorities report relay and voting status better: make it
+      easy to answer, "Why is my server not listed/not Guard/not
+      Running/etc"
+      o Write proposal
+      - Possibly implement, depending on discussion
+    - 162: Have consensuses come in multiple "flavours".
+      o Write proposal
+      - Possibly implement, depending on discussion.
+
+  - Needs a proposal, or at least some design
+    - Weaken the requirements for being a Guard, based on K's
+      measurements.
+K     - Finish measurements
+K?    - Write proposal
+    - Adaptive timeouts for giving up on circuits and streams.
+M     - Revise proposal 151
+    - Downweight guards more sensibly: be more forgiving about using
+      Guard nodes as non-first-hop.
+      - Write proposal.
+    - Lagged weight updates in consensuses: don't just move abruptly.
+M?    - Write proposal
+    d Don't kill a circuit on the first failed extend.
+
+- Installers
+  - Switch to MSI on win32
+  - Use Thandy, perhaps?

 - Deprecations
  - Make .exit safe, or make it off-by-default.
--- a/doc/TODO.external
+++ b/doc/TODO.external
@ -1,196 +1,4 @@
-$Id$
-Legend:
-SPEC!!  - Not specified
-SPEC    - Spec not finalized
-N       - nick claims
-R       - arma claims
-P       - phobos claims
-S       - Steven claims
-E       - Matt claims
-M       - Mike claims
-J       - Jeff claims
-I       - ioerror claims
-W       - weasel claims
-K       - Karsten claims
-C       - coderman claims
-        - Not done
-        * Top priority
-        . Partially done
-        o Done
-        d Deferrable
-        D Deferred
-        X Abandoned

-=======================================================================
-
-External constraints:
-
-For June/July:
-NR  - Work more on Paul's NRL research problem.
-
-For March 22:
-I   * Email auto-responder
-      * teach gettor how to ask for (and attach) split files.
-
-K   . Metrics.
-      . With Mike's help, use Torflow to start doing monthly rudimentary
-        performance evaluations:
-        . Circuit throughput and latency
-        - Measure via Broadband and dialup
-      . Publish a report addressing key long-term metrics questions:
-        . What metrics should we present?
-        . What data are available for these metrics?
-        . What data are missing, and can collect them safely? Can we
-          publish them safely?
-        . What systems are available to present this data?
-
-E   . Vidalia improvements
-      o Vidalia displays by-country user summary for bridge operators
-?       - write a help page for vidalia, "what is this"
-
-For mid August:
-
-Section 0, items that didn't make it into the original roadmap:
-
-0.1, installers and packaging
-C . i18n for the msi bundle files
-P . more consistent TBB builds
-IC- get a buildbot up again. Have Linux and BSD build machines.
-    (Windows would be nice but realistically will come later.)
-E - Get Tor to work properly on the iPhone.
-
-3.1, performance work. [Section numbers in here are from performance.pdf]
-  - High-priority items from performance.pdf
-RS  - 1.2, new circuit window sizes. make the default package window lower.
-R+  - 2.1, squeeze loud circuits
-      - Evaluate the code to see what stats we can keep about circuit use.
-      - Write proposals for various meddling. Look at the research papers
-        that Juliusz pointed us to. Ask our systems friends. Plan to put
-        a lot of the parameters in the consensus, so we can tune it with
-        short turnaround times.
-E+  - 2.5, Change Vidalia's default exit policy to not click "other
-      protocols". Or choose not to. Think this through first.
-R+  - 2.6, Tell users not to file-share.
-      - Put statement on the Tor front page
-      - Put statement on the download pages too
-      - And the FAQ
-    - 3.1.2, Tor weather
-I     - Implement time-to-notification (immediate, a day, a week)
-I     - Get a relay operator mailing list going, with a plan and supporting
-        scripts and so on.
-R     - Link to them from the Tor relay page
-R     - and the torrc.sample?
-SM  - 4.1, balance traffic better
-      - Steven and Mike should decide if we should do Steven's plan
-        (rejigger the bandwidth numbers at the authorities based on
-        Steven's algorithm), or Mike's plan (relay scanning to identify
-        the unbalanced relays and fix them on the fly), or both.
-      - Figure out how to actually modify bandwidths in the consensus. We
-        may need to change the consensus voting algorithm to decide what
-        bandwidth to advertise based on something other than median:
-        if 7 authorities provide bandwidths, and 2 are doing scanning,
-        then the 5 that aren't scanning will outvote any changes. Should
-        all 7 scan? Should only some vote? Extra points if it doesn't
-        change all the numbers every new consensus, so consensus diffing
-        is still practical.
-?   - 4.5, Older entry guards are overloaded
-      - Pick a conservative timeout like a month, and implement.
-M   - 5.2, better timeouts for giving up on circuits/streams
-      - clients gather data about circuit timeouts, and then abandon
-        circuits that take more than a std dev above that.
-
-4.1, IOCP / libevent / windows / tor
-N - get it working for nick
-N - put out a release so other people can start testing it.
-N - both the libevent buffer abstraction, and the
-    tor-uses-libevent-buffer-abstraction. Unless we think that's
-    unreachable for this milestone?
-
-4.2.1, risks from becoming a relay
-S - Have a clear plan for how users who become relays will be safe,
-    and be confident that we can build this plan.
-    - evaluate all the various attacks that are made possible by relaying.
-      specifically, see "relaying-traffic attacks" in 6.6.
-    - identify and evaluate ways to make them not a big deal
-      - setting a low RelayBandwidth
-      - Nick Hopper's FC08 paper suggesting that we should do a modified
-        round-robin so we leak less about other circuits
-      - instructing clients to disable pings in their firewall, etc
-    - pick the promising ones, improve them so they're even better, and
-      spec them out so we know how to build them and how much effort is
-      involved in building them.
-
-4.5, clients download less directory info
-N * deploy proposal 158.
-N - decide whether to do proposal 140. if so, construct an implementation
-    plan for how we'll do it. if not, explain why not.
-
-5.1, Normalize TLS fingerprint
-N o write a draft list of possible attacks for this section, with
-    estimates about difficulty of attack, difficulty of solution, etc
-N - revisit the list and revise our plans as needed
-NR- put up a blog post about the two contradictory conclusions: we can
-    discuss the theory of arms races, and our quandry, without revealing
-    any specific vulnerabilities. (or decide not to put up a blog post,
-    and explain why not.)
-
-5.5, email autoresponder
-I . maintenance and keeping it running
-
-5.7.2, metrics
-
-XXX.
-
-6.2, Vidalia work
-E - add breakpad support or similar for windows debugging
-E o let vidalia change languages without needing a restart
-E - Implement the status warning event interface started for the
-    phase one deliverables.
-E - Work with Steve Tyree on building a Vidalia plugin API to enable
-    building Herdict and TBB plugins.
-
-6.3, Node scanning
-M - Steps toward automation
-    - Set up email list for results
-    - Map failure types to potential BadExit lines
-M - Improve the ability of SoaT to mimic various real web browsers
-    - randomizing user agents and locale strings
-    - caching, XMLHTTPRequest, form posting, content sniffing
-    - Investigate ideas like running Chrome/xulrunner in parallel
-M - Other protocols
-    - SSH, IMAPS, POPS, SMTPS
-M - Add ability to geolocalize exit selection based on scanner location
-    - Use this to rescan dynamic urls filtered by the URL filter
-
-6.4, Torbutton development
-M - Resolve extension conflicts and other high priority bugs
-M - Fix or hack around ugly firefox bugs, especially Timezone issue.
-    Definitely leaning towards "hack around" unless we see some
-    level of love from Mozilla.
-M - Vidalia New Nym Integration
-    - Implement for Torbutton to pick up on Vidalia's NEWNYM and clear
-      cookies based on FoeBud's source
-    - Do this in such a way that we could adapt polipo to purge cache
-      if we were so inclined
-M - Write up a summary of our options for dealing with the google
-    you-must-solve-a-captcha-to-search problem, and pick one as our
-    favorite option.
-
-6.6, Evaluate new anonymity attacks
-S - relaying-traffic attacks
-    - original murdoch-danezis attack
-    - nick hopper's latency measurement attack
-    - columbia bandwidth measurement attack
-    - christian grothoff's long-circuit attack
-S - client attacks
-    - website fingerprinting
-
-7.1, Tor VM Research, analysis, and prototyping
-C . Get a working package out, meaning other people are testing it.
-
-7.2, Tor Browser Bundle
-I - Port to one of OS X or Linux, and start the port to the other.
-I . Make it the recommended Tor download on Windows
-I - Make sure it's easy to un-brand TBB in case Firefox asks us to
-I - Evaluate CCC's Freedom Stick
+[This file moved to svn in /projects/todo/. More people can edit
+it more easily there. -RD]

--- a/doc/TODO.future
+++ b/doc/TODO.future
@ -1,4 +1,3 @@
-$Id$
 Legend:
 SPEC!!  - Not specified
 SPEC    - Spec not finalized
--- a/doc/design-paper/latex8.bst
+++ b/doc/design-paper/latex8.bst
@ -1,8 +1,6 @@

 % ---------------------------------------------------------------
 %
-% $Id$
-%
 % by Paolo.Ienne@di.epfl.ch
 %

--- a/doc/design-paper/usenix.sty
+++ b/doc/design-paper/usenix.sty
@ -5,8 +5,6 @@
 %    \usepackage{usenix-2e}
 % and put {\rm ....} around the author names.
 %
-% $Id$
-%
 % The following definitions are modifications of standard article.sty
 % definitions, arranged to do a better job of matching the USENIX
 % guidelines.
--- a/doc/spec/address-spec.txt
+++ b/doc/spec/address-spec.txt
@ -1,4 +1,3 @@
-$Id$

                          Special Hostnames in Tor
                               Nick Mathewson
@ -34,10 +33,13 @@ $Id$
  "www.google.com.foo.exit=64.233.161.99.foo.exit" to speed subsequent
  lookups.

+  The .exit notation is disabled by default as of Tor 0.2.2.1-alpha, due
+  to potential application-level attacks.
+
  EXAMPLES:
     www.example.com.exampletornode.exit

-        Connect to www.example.com from the node called "exampletornode."
+        Connect to www.example.com from the node called "exampletornode".

     exampletornode.exit

@ -54,15 +56,3 @@ $Id$
  When Tor sees an address in this format, it tries to look up and connect to
  the specified hidden service.  See rend-spec.txt for full details.

-4. .noconnect
-
-  SYNTAX:  [string].noconnect
-
-  When Tor sees an address in this format, it immediately closes the
-  connection without attaching it to any circuit.  This is useful for
-  controllers that want to test whether a given application is indeed using
-  the same instance of Tor that they're controlling.
-
-5. [XXX Is there a ".virtual" address that we expose too, or is that
-just intended to be internal? -RD]
-
--- a/doc/spec/bridges-spec.txt
+++ b/doc/spec/bridges-spec.txt
@ -1,4 +1,3 @@
-$Id$

                          Tor bridges specification

--- a/doc/spec/control-spec-v0.txt
+++ b/doc/spec/control-spec-v0.txt
@ -1,4 +1,3 @@
-$Id$

                   TC: A Tor control protocol (Version 0)

--- a/doc/spec/control-spec.txt
+++ b/doc/spec/control-spec.txt
@ -1,4 +1,3 @@
-$Id$

                   TC: A Tor control protocol (Version 1)

@ -220,7 +219,7 @@ $Id$
         "INFO" / "NOTICE" / "WARN" / "ERR" / "NEWDESC" / "ADDRMAP" /
         "AUTHDIR_NEWDESCS" / "DESCCHANGED" / "STATUS_GENERAL" /
         "STATUS_CLIENT" / "STATUS_SERVER" / "GUARD" / "NS" / "STREAM_BW" /
-         "CLIENTS_SEEN"
+         "CLIENTS_SEEN" / "NEWCONSENSUS"

  Any events *not* listed in the SETEVENTS line are turned off; thus, sending
  SETEVENTS with an empty body turns off all event reporting.
@ -503,7 +502,7 @@ $Id$
      start and the rest of the interval respectively.  The 'interval-start'
      and 'interval-end' fields are the borders of the current interval; the
      'interval-wake' field is the time within the current interval (if any)
-      where we plan[ned] to start being active.
+      where we plan[ned] to start being active. The times are GMT.

    "config/names"
      A series of lines listing the available configuration options. Each is
@ -563,14 +562,14 @@ $Id$
      states. See Section 4.1.10 for explanations. (Only a few of the
      status events are available as getinfo's currently. Let us know if
      you want more exposed.)
-    "status/reachability/or"
+    "status/reachability-succeeded/or"
      0 or 1, depending on whether we've found our ORPort reachable.
-    "status/reachability/dir"
+    "status/reachability-succeeded/dir"
      0 or 1, depending on whether we've found our DirPort reachable.
-    "status/reachability"
+    "status/reachability-succeeded"
      "OR=" ("0"/"1") SP "DIR=" ("0"/"1")
-      Combines status/reachability/*; controllers MUST ignore unrecognized
-      elements in this entry.
+      Combines status/reachability-succeeded/*; controllers MUST ignore
+      unrecognized elements in this entry.
    "status/bootstrap-phase"
      Returns the most recent bootstrap phase status event
      sent. Specifically, it returns a string starting with either
@ -774,9 +773,8 @@ $Id$
     Same as passing 'EXTENDED' to SETEVENTS; this is the preferred way to
     request the extended event syntax.

-     This will not be always-enabled until at least two stable releases
-     after 0.1.2.3-alpha, the release where it was first used for
-     anything.
+    This feature was first used in 0.1.2.3-alpha.  It is always-on in
+    Tor 0.2.2.1-alpha and later.

  VERBOSE_NAMES

@ -787,8 +785,9 @@ $Id$
     LongName format includes a Fingerprint, an indication of Named status,
     and a Nickname (if one is known).

-     This will not be always-enabled until at least two stable releases
-     after 0.1.2.2-alpha, the release where it was first available.
+     This will not be always-enabled until at least two stable
+     releases after 0.1.2.2-alpha, the release where it was first
+     available.  It is always-on in Tor 0.2.2.1-alpha and later.

 3.20. RESOLVE

--- a/doc/spec/dir-spec-v1.txt
+++ b/doc/spec/dir-spec-v1.txt
@ -1,4 +1,3 @@
-$Id$

                         Tor Protocol Specification

--- a/doc/spec/dir-spec-v2.txt
+++ b/doc/spec/dir-spec-v2.txt
@ -1,4 +1,3 @@
-$Id$

                      Tor directory protocol, version 2

--- a/doc/spec/dir-spec.txt
+++ b/doc/spec/dir-spec.txt
@ -1,4 +1,3 @@
-$Id$

                      Tor directory protocol, version 3

@ -594,7 +593,7 @@ $Id$

   "allow-single-hop-exits"

-       [At most one.]
+       [At most once.]

       Present only if the router allows single-hop circuits to make exit
       connections.  Most Tor servers do not support this: this is
@ -642,6 +641,200 @@ $Id$
        "geoip-start" is the time at which we began collecting geoip
        statistics.

+    "dirreq-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        A "dirreq-stats-end" line, as well as any other "dirreq-*" line,
+        is only added when the relay has opened its Dir port and after 24
+        hours of measuring directory requests.
+
+    "dirreq-v2-ips" CC=N,CC=N,... NL
+        [At most once.]
+    "dirreq-v3-ips" CC=N,CC=N,... NL
+        [At most once.]
+
+        List of mappings from two-letter country codes to the number of
+        unique IP addresses that have connected from that country to
+        request a v2/v3 network status, rounded up to the nearest multiple
+        of 8. Only those IP addresses are counted that the directory can
+        answer with a 200 OK status code.
+
+    "dirreq-v2-reqs" CC=N,CC=N,... NL
+        [At most once.]
+    "dirreq-v3-reqs" CC=N,CC=N,... NL
+        [At most once.]
+
+        List of mappings from two-letter country codes to the number of
+        requests for v2/v3 network statuses from that country, rounded up
+        to the nearest multiple of 8. Only those requests are counted that
+        the directory can answer with a 200 OK status code.
+
+    "dirreq-v2-share" num% NL
+        [At most once.]
+    "dirreq-v3-share" num% NL
+        [At most once.]
+
+        The share of v2/v3 network status requests that the directory
+        expects to receive from clients based on its advertised bandwidth
+        compared to the overall network bandwidth capacity. Shares are
+        formatted in percent with two decimal places. Shares are
+        calculated as means over the whole 24-hour interval.
+
+    "dirreq-v2-resp" status=num,... NL
+        [At most once.]
+    "dirreq-v3-resp" status=nul,... NL
+        [At most once.]
+
+        List of mappings from response statuses to the number of requests
+        for v2/v3 network statuses that were answered with that response
+        status, rounded up to the nearest multiple of 4. Only response
+        statuses with at least 1 response are reported. New response
+        statuses can be added at any time. The current list of response
+        statuses is as follows:
+
+        "ok": a network status request is answered; this number
+           corresponds to the sum of all requests as reported in
+           "dirreq-v2-reqs" or "dirreq-v3-reqs", respectively, before
+           rounding up.
+        "not-enough-sigs: a version 3 network status is not signed by a
+           sufficient number of requested authorities.
+        "unavailable": a requested network status object is unavailable.
+        "not-found": a requested network status is not found.
+        "not-modified": a network status has not been modified since the
+           If-Modified-Since time that is included in the request.
+        "busy": the directory is busy.
+
+    "dirreq-v2-direct-dl" key=val,... NL
+        [At most once.]
+    "dirreq-v3-direct-dl" key=val,... NL
+        [At most once.]
+    "dirreq-v2-tunneled-dl" key=val,... NL
+        [At most once.]
+    "dirreq-v3-tunneled-dl" key=val,... NL
+        [At most once.]
+
+        List of statistics about possible failures in the download process
+        of v2/v3 network statuses. Requests are either "direct"
+        HTTP-encoded requests over the relay's directory port, or
+        "tunneled" requests using a BEGIN_DIR cell over the relay's OR
+        port. The list of possible statistics can change, and statistics
+        can be left out from reporting. The current list of statistics is
+        as follows:
+
+        Successful downloads and failures:
+
+        "complete": a client has finished the download successfully.
+        "timeout": a download did not finish within 10 minutes after
+           starting to send the response.
+        "running": a download is still running at the end of the
+           measurement period for less than 10 minutes after starting to
+           send the response.
+
+        Download times:
+
+        "min", "max": smallest and largest measured bandwidth in B/s.
+        "d[1-4,6-9]": 1st to 4th and 6th to 9th decile of measured
+           bandwidth in B/s. For a given decile i, i/10 of all downloads
+           had a smaller bandwidth than di, and (10-i)/10 of all downloads
+           had a larger bandwidth than di.
+        "q[1,3]": 1st and 3rd quartile of measured bandwidth in B/s. One
+           fourth of all downloads had a smaller bandwidth than q1, one
+           fourth of all downloads had a larger bandwidth than q3, and the
+           remaining half of all downloads had a bandwidth between q1 and
+           q3.
+        "md": median of measured bandwidth in B/s. Half of the downloads
+           had a smaller bandwidth than md, the other half had a larger
+           bandwidth than md.
+
+    "entry-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        An "entry-stats-end" line, as well as any other "entry-*"
+        line, is first added after the relay has been running for at least
+        24 hours.
+
+    "entry-ips" CC=N,CC=N,... NL
+        [At most once.]
+
+        List of mappings from two-letter country codes to the number of
+        unique IP addresses that have connected from that country to the
+        relay and which are no known other relays, rounded up to the
+        nearest multiple of 8.
+
+    "cell-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        A "cell-stats-end" line, as well as any other "cell-*" line,
+        is first added after the relay has been running for at least 24
+        hours.
+
+    "cell-processed-cells" num,...,num NL
+        [At most once.]
+
+        Mean number of processed cells per circuit, subdivided into
+        deciles of circuits by the number of cells they have processed in
+        descending order from loudest to quietest circuits.
+
+    "cell-queued-cells" num,...,num NL
+        [At most once.]
+
+        Mean number of cells contained in queues by circuit decile. These
+        means are calculated by 1) determining the mean number of cells in
+        a single circuit between its creation and its termination and 2)
+        calculating the mean for all circuits in a given decile as
+        determined in "cell-processed-cells". Numbers have a precision of
+        two decimal places.
+
+    "cell-time-in-queue" num,...,num NL
+        [At most once.]
+
+        Mean time cells spend in circuit queues in milliseconds. Times are
+        calculated by 1) determining the mean time cells spend in the
+        queue of a single circuit and 2) calculating the mean for all
+        circuits in a given decile as determined in
+        "cell-processed-cells".
+
+    "cell-circuits-per-decile" num NL
+        [At most once.]
+
+        Mean number of circuits that are included in any of the deciles,
+        rounded up to the next integer.
+
+    "exit-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+        [At most once.]
+
+        YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+        interval of length NSEC seconds (86400 seconds by default).
+
+        An "exit-stats-end" line, as well as any other "exit-*" line, is
+        first added after the relay has been running for at least 24 hours
+        and only if the relay permits exiting (where exiting to a single
+        port and IP address is sufficient).
+
+    "exit-kibibytes-written" port=N,port=N,... NL
+        [At most once.]
+    "exit-kibibytes-read" port=N,port=N,... NL
+        [At most once.]
+
+        List of mappings from ports to the number of kibibytes that the
+        relay has written to or read from exit connections to that port,
+        rounded up to the next full kibibyte.
+
+    "exit-streams-opened" port=N,port=N,... NL
+        [At most once.]
+
+        List of mappings from ports to the number of opened exit streams
+        to that port, rounded up to the nearest multiple of 4.
+
    "router-signature" NL Signature NL
        [At end, exactly once.]

@ -798,7 +991,7 @@ $Id$
   documents are described in section XXX below.

   Status documents contain a preamble, an authority section, a list of
-   router status entries, and one more footers signature, in that order.
+   router status entries, and one or more footer signature, in that order.

   Unlike other formats described above, a SP in these documents must be a
   single space character (hex 20).
@ -1030,13 +1223,20 @@ $Id$
        descriptors if they would cause "v" lines to be over 128 characters
        long.

-    "w" SP "Bandwidth=" INT NL
+    "w" SP "Bandwidth=" INT [SP "Measured=" INT] NL

        [At most once.]

        An estimate of the bandwidth of this server, in an arbitrary
        unit (currently kilobytes per second).  Used to weight router
-        selection.  Other weighting keywords may be added later.
+        selection. 
+
+        Additionally, the Measured= keyword is present in votes by 
+        participating bandwidth measurement authorites to indicate
+        a measured bandwidth currently produced by measuring stream 
+        capacities. 
+
+        Other weighting keywords may be added later.
        Clients MUST ignore keywords they do not recognize.

    "p" SP ("accept" / "reject") SP PortList NL
@ -1179,6 +1379,13 @@ $Id$
   rate limit from the router descriptor.  It is given in kilobytes
   per second, and capped at some arbitrary value (currently 10 MB/s).

+   The Measured= keyword on a "w" line vote is currently computed
+   by multiplying the previous published consensus bandwidth by the 
+   ratio of the measured average node stream capacity to the network 
+   average. If 3 or more authorities provide a Measured= keyword for 
+   a router, the authorites produce a consensus containing a "w" 
+   Bandwidth= keyword equal to the median of the Measured= votes.
+
   The ports listed in a "p" line should be taken as those ports for
   which the router's exit policy permits 'most' addresses, ignoring any
   accept not for all addresses, ignoring all rejects for private
@ -1261,6 +1468,11 @@ $Id$
          one, breaking ties in favor of the lexicographically larger
          vote.)  The port list is encoded as specified in 3.4.2.

+        * If consensus-method 6 or later is in use and if 3 or more 
+          authorities provide a Measured= keyword in their votes for 
+          a router, the authorities produce a consensus containing a 
+          Bandwidth= keyword equal to the median of the Measured= votes.
+
     The signatures at the end of a consensus document are sorted in
     ascending order by identity digest.

@ -1281,6 +1493,7 @@ $Id$
     "3" -- Added legacy ID key support to aid in authority ID key rollovers
     "4" -- No longer list routers that are not running in the consensus
     "5" -- adds support for "w" and "p" lines.
+     "6" -- Prefers measured bandwidth values rather than advertised

   Before generating a consensus, an authority must decide which consensus
   method to use.  To do this, it looks for the highest version number
--- a/doc/spec/path-spec.txt
+++ b/doc/spec/path-spec.txt
@ -1,4 +1,3 @@
-$Id$

                           Tor Path Specification

@ -72,6 +71,24 @@ of their choices.
   is unknown (usually its target IP), but we believe the path probably
   supports the request according to the rules given below.

+1.1. A server's bandwidth
+
+   Old versions of Tor did not report bandwidths in network status
+   documents, so clients had to learn them from the routers' advertised
+   server descriptors.
+
+   For versions of Tor prior to 0.2.1.17-rc, everywhere below where we
+   refer to a server's "bandwidth", we mean its clipped advertised
+   bandwidth, computed by taking the smaller of the 'rate' and
+   'observed' arguments to the "bandwidth" element in the server's
+   descriptor.  If a router's advertised bandwidth is greater than
+   MAX_BELIEVABLE_BANDWIDTH (currently 10 MB/s), we clipped to that
+   value.
+
+   For more recent versions of Tor, we take the bandwidth value declared
+   in the consensus, and fall back to the clipped advertised bandwidth
+   only if the consensus does not have bandwidths listed.
+
 2. Building circuits

 2.1. When we build
@ -179,16 +196,13 @@ of their choices.
   multiple candidates for a path element, we choose randomly.

   For "fast" circuits, we pick a given router as an exit with probability
-   proportional to its advertised bandwidth [the smaller of the 'rate' and
-   'observed' arguments to the "bandwidth" element in its descriptor].  If a
-   router's advertised bandwidth is greater than MAX_BELIEVABLE_BANDWIDTH
-   (currently 10 MB/s), we clip to that value.
+   proportional to its bandwidth.

   For non-exit positions on "fast" circuits, we pick routers as above, but
-   we weight the clipped advertised bandwidth of Exit-flagged nodes depending
+   we weight the bandwidth of Exit-flagged nodes depending
   on the fraction of bandwidth available from non-Exit nodes.  Call the
-   total clipped advertised bandwidth for Exit nodes under consideration E,
-   and the total clipped advertised bandwidth for all nodes under
+   total bandwidth for Exit nodes under consideration E,
+   and the total bandwidth for all nodes under
   consideration T.  If E<T/3, we do not consider Exit-flagged nodes.
   Otherwise, we weight their bandwidth with the factor (E-T/3)/E. This 
   ensures that bandwidth is evenly distributed over nodes in 3-hop paths.
@ -306,7 +320,7 @@ of their choices.
  We use Guard nodes (also called "helper nodes" in the literature) to
  prevent certain profiling attacks.  Here's the risk: if we choose entry and
  exit nodes at random, and an attacker controls C out of N servers
-  (ignoring advertised bandwidth), then the
+  (ignoring bandwidth), then the
  attacker will control the entry and exit node of any given circuit with
  probability (C/N)^2.  But as we make many different circuits over time,
  then the probability that the attacker will see a sample of about (C/N)^2
--- a/doc/spec/proposals/000-index.txt
+++ b/doc/spec/proposals/000-index.txt
@ -1,7 +1,5 @@
 Filename: 000-index.txt
 Title: Index of Tor Proposals
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 26-Jan-2007
 Status: Meta
@ -56,7 +54,7 @@ Proposals by number:
 131  Help users to verify they are using Tor [NEEDS-REVISION]
 132  A Tor Web Service For Verifying Correct Browser Configuration [DRAFT]
 133  Incorporate Unreachable ORs into the Tor Network [DRAFT]
-134  More robust consensus voting with diverse authority sets [ACCEPTED]
+134  More robust consensus voting with diverse authority sets [REJECTED]
 135  Simplify Configuration of Private Tor Networks [CLOSED]
 136  Mass authority migration with legacy keys [CLOSED]
 137  Keep controllers informed as Tor bootstraps [CLOSED]
@ -82,6 +80,13 @@ Proposals by number:
 157  Make certificate downloads specific [ACCEPTED]
 158  Clients download consensus + microdescriptors [OPEN]
 159  Exit Scanning [OPEN]
+160  Authorities vote for bandwidth offsets in consensus [OPEN]
+161  Computing Bandwidth Adjustments [OPEN]
+162  Publish the consensus in multiple flavors [OPEN]
+163  Detecting whether a connection comes from a client [OPEN]
+164  Reporting the status of server votes [OPEN]
+165  Easy migration for voting authority sets [OPEN]
+166  Including Network Statistics in Extra-Info Documents [ACCEPTED]


 Proposals by status:
@ -103,14 +108,20 @@ Proposals by status:
   156  Tracking blocked ports on the client side [for 0.2.?]
   158  Clients download consensus + microdescriptors
   159  Exit Scanning
+   160  Authorities vote for bandwidth offsets in consensus [for 0.2.2.x]
+   161  Computing Bandwidth Adjustments [for 0.2.2.x]
+   162  Publish the consensus in multiple flavors [for 0.2.2]
+   163  Detecting whether a connection comes from a client [for 0.2.2]
+   164  Reporting the status of server votes [for 0.2.2]
+   165  Easy migration for voting authority sets
 ACCEPTED:
   110  Avoiding infinite length circuits [for 0.2.1.x] [in 0.2.1.3-alpha]
   117  IPv6 exits [for 0.2.1.x]
   118  Advertising multiple ORPorts at once [for 0.2.1.x]
-   134  More robust consensus voting with diverse authority sets [for 0.2.2.x]
   140  Provide diffs between consensuses [for 0.2.2.x]
   147  Eliminate the need for v2 directories in generating v3 directories [for 0.2.1.x]
   157  Make certificate downloads specific [for 0.2.1.x]
+   166  Including Network Statistics in Extra-Info Documents [for 0.2.2]
 META:
   000  Index of Tor Proposals
   001  The Tor Proposal Process
@ -159,3 +170,5 @@ Proposals by status:
   120  Shutdown descriptors when Tor servers stop
   128  Families of private bridges
   142  Combine Introduction and Rendezvous Points
+ REJECTED:
+   134  More robust consensus voting with diverse authority sets
--- a/doc/spec/proposals/001-process.txt
+++ b/doc/spec/proposals/001-process.txt
@ -1,7 +1,5 @@
 Filename: 001-process.txt
 Title: The Tor Proposal Process
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 30-Jan-2007
 Status: Meta
@ -47,7 +45,7 @@ How to change the specs now:
   Like an RFC, every proposal gets a number.  Unlike RFCs, proposals can
   change over time and keep the same number, until they are finally
   accepted or rejected.  The history for each proposal
-   will be stored in the Tor Subversion repository.
+   will be stored in the Tor repository.

   Once a proposal is in the repository, we should discuss and improve it
   until we've reached consensus that it's a good idea, and that it's
@ -82,9 +80,7 @@ How new proposals get added:
 What should go in a proposal:

   Every proposal should have a header containing these fields:
-     Filename, Title, Version, Last-Modified, Author, Created, Status.
-   The Version and Last-Modified fields should use the SVN Revision and Date
-   tags respectively.
+     Filename, Title, Author, Created, Status.

   These fields are optional but recommended:
     Target, Implemented-In.
@ -97,7 +93,7 @@ What should go in a proposal:
   what the proposal's about, what it does, and about what state it's in.

   After the Overview, the proposal becomes more free-form.  Depending on its
-   the length and complexity, the proposal can break into sections as
+   length and complexity, the proposal can break into sections as
   appropriate, or follow a short discursive format.  Every proposal should
   contain at least the following information before it is "ACCEPTED",
   though the information does not need to be in sections with these names.
--- a/doc/spec/proposals/098-todo.txt
+++ b/doc/spec/proposals/098-todo.txt
@ -1,7 +1,5 @@
 Filename: 098-todo.txt
 Title: Proposals that should be written
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson, Roger Dingledine
 Created: 26-Jan-2007
 Status: Meta
--- a/doc/spec/proposals/099-misc.txt
+++ b/doc/spec/proposals/099-misc.txt
@ -1,7 +1,5 @@
 Filename: 099-misc.txt
 Title: Miscellaneous proposals
-Version: $Revision$
-Last-Modified: $Date$
 Author: Various
 Created: 26-Jan-2007
 Status: Meta
--- a/doc/spec/proposals/100-tor-spec-udp.txt
+++ b/doc/spec/proposals/100-tor-spec-udp.txt
@ -1,7 +1,5 @@
 Filename: 100-tor-spec-udp.txt
 Title: Tor Unreliable Datagram Extension Proposal
-Version: $Revision$
-Last-Modified: $Date$
 Author: Marc Liberatore
 Created: 23 Feb 2006
 Status: Dead
--- a/doc/spec/proposals/101-dir-voting.txt
+++ b/doc/spec/proposals/101-dir-voting.txt
@ -1,7 +1,5 @@
 Filename: 101-dir-voting.txt
 Title: Voting on the Tor Directory System
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: Nov 2006
 Status: Closed
--- a/doc/spec/proposals/102-drop-opt.txt
+++ b/doc/spec/proposals/102-drop-opt.txt
@ -1,7 +1,5 @@
 Filename: 102-drop-opt.txt
 Title: Dropping "opt" from the directory format
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: Jan 2007
 Status: Closed
--- a/doc/spec/proposals/103-multilevel-keys.txt
+++ b/doc/spec/proposals/103-multilevel-keys.txt
@ -1,7 +1,5 @@
 Filename: 103-multilevel-keys.txt
 Title: Splitting identity key from regularly used signing key.
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: Jan 2007
 Status: Closed
--- a/doc/spec/proposals/104-short-descriptors.txt
+++ b/doc/spec/proposals/104-short-descriptors.txt
@ -1,7 +1,5 @@
 Filename: 104-short-descriptors.txt
 Title: Long and Short Router Descriptors
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: Jan 2007
 Status: Closed
--- a/doc/spec/proposals/105-handshake-revision.txt
+++ b/doc/spec/proposals/105-handshake-revision.txt
@ -1,7 +1,5 @@
 Filename: 105-handshake-revision.txt
 Title: Version negotiation for the Tor protocol.
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson, Roger Dingledine
 Created: Jan 2007
 Status: Closed
--- a/doc/spec/proposals/106-less-tls-constraint.txt
+++ b/doc/spec/proposals/106-less-tls-constraint.txt
@ -1,7 +1,5 @@
 Filename: 106-less-tls-constraint.txt
 Title: Checking fewer things during TLS handshakes
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 9-Feb-2007
 Status: Closed
--- a/doc/spec/proposals/107-uptime-sanity-checking.txt
+++ b/doc/spec/proposals/107-uptime-sanity-checking.txt
@ -1,7 +1,5 @@
 Filename: 107-uptime-sanity-checking.txt
 Title: Uptime Sanity Checking
-Version: $Revision$
-Last-Modified: $Date$
 Author: Kevin Bauer & Damon McCoy
 Created: 8-March-2007
 Status: Closed
--- a/doc/spec/proposals/108-mtbf-based-stability.txt
+++ b/doc/spec/proposals/108-mtbf-based-stability.txt
@ -1,7 +1,5 @@
 Filename: 108-mtbf-based-stability.txt
 Title: Base "Stable" Flag on Mean Time Between Failures
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 10-Mar-2007
 Status: Closed
--- a/doc/spec/proposals/109-no-sharing-ips.txt
+++ b/doc/spec/proposals/109-no-sharing-ips.txt
@ -1,7 +1,5 @@
 Filename: 109-no-sharing-ips.txt
 Title: No more than one server per IP address.
-Version: $Revision$
-Last-Modified: $Date$
 Author: Kevin Bauer & Damon McCoy
 Created: 9-March-2007
 Status: Closed
--- a/doc/spec/proposals/110-avoid-infinite-circuits.txt
+++ b/doc/spec/proposals/110-avoid-infinite-circuits.txt
@ -1,7 +1,5 @@
 Filename: 110-avoid-infinite-circuits.txt
 Title: Avoiding infinite length circuits
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 13-Mar-2007
 Status: Accepted
--- a/doc/spec/proposals/111-local-traffic-priority.txt
+++ b/doc/spec/proposals/111-local-traffic-priority.txt
@ -1,7 +1,5 @@
 Filename: 111-local-traffic-priority.txt
 Title: Prioritizing local traffic over relayed traffic
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 14-Mar-2007
 Status: Closed
--- a/doc/spec/proposals/112-bring-back-pathlencoinweight.txt
+++ b/doc/spec/proposals/112-bring-back-pathlencoinweight.txt
@ -1,7 +1,5 @@
 Filename: 112-bring-back-pathlencoinweight.txt
 Title: Bring Back Pathlen Coin Weight
-Version: $Revision$
-Last-Modified: $Date$
 Author: Mike Perry
 Created:
 Status: Superseded
--- a/doc/spec/proposals/113-fast-authority-interface.txt
+++ b/doc/spec/proposals/113-fast-authority-interface.txt
@ -1,7 +1,5 @@
 Filename: 113-fast-authority-interface.txt
 Title: Simplifying directory authority administration
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created:
 Status: Superseded
--- a/doc/spec/proposals/114-distributed-storage.txt
+++ b/doc/spec/proposals/114-distributed-storage.txt
@ -1,7 +1,5 @@
 Filename: 114-distributed-storage.txt
 Title: Distributed Storage for Tor Hidden Service Descriptors
-Version: $Revision$
-Last-Modified: $Date$
 Author: Karsten Loesing
 Created: 13-May-2007
 Status: Closed
--- a/doc/spec/proposals/115-two-hop-paths.txt
+++ b/doc/spec/proposals/115-two-hop-paths.txt
@ -1,7 +1,5 @@
 Filename: 115-two-hop-paths.txt
 Title: Two Hop Paths
-Version: $Revision$
-Last-Modified: $Date$
 Author: Mike Perry
 Created:
 Status: Dead
--- a/doc/spec/proposals/116-two-hop-paths-from-guard.txt
+++ b/doc/spec/proposals/116-two-hop-paths-from-guard.txt
@ -1,7 +1,5 @@
 Filename: 116-two-hop-paths-from-guard.txt
 Title: Two hop paths from entry guards
-Version: $Revision$
-Last-Modified: $Date$
 Author: Michael Lieberman
 Created: 26-Jun-2007
 Status: Dead
--- a/doc/spec/proposals/117-ipv6-exits.txt
+++ b/doc/spec/proposals/117-ipv6-exits.txt
@ -1,7 +1,5 @@
 Filename: 117-ipv6-exits.txt
 Title: IPv6 exits
-Version: $Revision$
-Last-Modified: $Date$
 Author: coderman
 Created: 10-Jul-2007
 Status: Accepted
--- a/doc/spec/proposals/118-multiple-orports.txt
+++ b/doc/spec/proposals/118-multiple-orports.txt
@ -1,7 +1,5 @@
 Filename: 118-multiple-orports.txt
 Title: Advertising multiple ORPorts at once
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 09-Jul-2007
 Status: Accepted
--- a/doc/spec/proposals/119-controlport-auth.txt
+++ b/doc/spec/proposals/119-controlport-auth.txt
@ -1,7 +1,5 @@
 Filename: 119-controlport-auth.txt
 Title: New PROTOCOLINFO command for controllers
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 14-Aug-2007
 Status: Closed
--- a/doc/spec/proposals/120-shutdown-descriptors.txt
+++ b/doc/spec/proposals/120-shutdown-descriptors.txt
@ -1,7 +1,5 @@
 Filename: 120-shutdown-descriptors.txt
 Title: Shutdown descriptors when Tor servers stop
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 15-Aug-2007
 Status: Dead
--- a/doc/spec/proposals/121-hidden-service-authentication.txt
+++ b/doc/spec/proposals/121-hidden-service-authentication.txt
@ -1,7 +1,5 @@
 Filename: 121-hidden-service-authentication.txt
 Title: Hidden Service Authentication
-Version: $Revision$
-Last-Modified: $Date$
 Author: Tobias Kamm, Thomas Lauterbach, Karsten Loesing, Ferdinand Rieger,
        Christoph Weingarten
 Created: 10-Sep-2007
--- a/doc/spec/proposals/122-unnamed-flag.txt
+++ b/doc/spec/proposals/122-unnamed-flag.txt
@ -1,7 +1,5 @@
 Filename: 122-unnamed-flag.txt
 Title: Network status entries need a new Unnamed flag
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 04-Oct-2007
 Status: Closed
--- a/doc/spec/proposals/123-autonaming.txt
+++ b/doc/spec/proposals/123-autonaming.txt
@ -1,7 +1,5 @@
 Filename: 123-autonaming.txt
 Title: Naming authorities automatically create bindings
-Version: $Revision$
-Last-Modified: $Date$
 Author: Peter Palfrader
 Created: 2007-10-11
 Status: Closed
--- a/doc/spec/proposals/124-tls-certificates.txt
+++ b/doc/spec/proposals/124-tls-certificates.txt
@ -1,7 +1,5 @@
 Filename: 124-tls-certificates.txt
 Title: Blocking resistant TLS certificate usage
-Version: $Revision$
-Last-Modified: $Date$
 Author: Steven J. Murdoch
 Created: 2007-10-25
 Status: Superseded
--- a/doc/spec/proposals/125-bridges.txt
+++ b/doc/spec/proposals/125-bridges.txt
@ -1,7 +1,5 @@
 Filename: 125-bridges.txt
 Title: Behavior for bridge users, bridge relays, and bridge authorities
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 11-Nov-2007
 Status: Closed
--- a/doc/spec/proposals/126-geoip-reporting.txt
+++ b/doc/spec/proposals/126-geoip-reporting.txt
@ -1,7 +1,5 @@
 Filename: 126-geoip-reporting.txt
 Title: Getting GeoIP data and publishing usage summaries
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 2007-11-24
 Status: Closed
--- a/doc/spec/proposals/127-dirport-mirrors-downloads.txt
+++ b/doc/spec/proposals/127-dirport-mirrors-downloads.txt
@ -1,7 +1,5 @@
 Filename: 127-dirport-mirrors-downloads.txt
 Title: Relaying dirport requests to Tor download site / website
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 2007-12-02
 Status: Draft
--- a/doc/spec/proposals/128-bridge-families.txt
+++ b/doc/spec/proposals/128-bridge-families.txt
@ -1,7 +1,5 @@
 Filename: 128-bridge-families.txt
 Title: Families of private bridges
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 2007-12-xx
 Status: Dead
--- a/doc/spec/proposals/129-reject-plaintext-ports.txt
+++ b/doc/spec/proposals/129-reject-plaintext-ports.txt
@ -1,7 +1,5 @@
 Filename: 129-reject-plaintext-ports.txt
 Title: Block Insecure Protocols by Default
-Version: $Revision$
-Last-Modified: $Date$
 Author: Kevin Bauer & Damon McCoy
 Created: 2008-01-15
 Status: Closed
--- a/doc/spec/proposals/130-v2-conn-protocol.txt
+++ b/doc/spec/proposals/130-v2-conn-protocol.txt
@ -1,7 +1,5 @@
 Filename: 130-v2-conn-protocol.txt
 Title: Version 2 Tor connection protocol
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 2007-10-25
 Status: Closed
--- a/doc/spec/proposals/131-verify-tor-usage.txt
+++ b/doc/spec/proposals/131-verify-tor-usage.txt
@ -1,7 +1,5 @@
 Filename: 131-verify-tor-usage.txt
 Title: Help users to verify they are using Tor
-Version: $Revision$
-Last-Modified: $Date$
 Author: Steven J. Murdoch
 Created: 2008-01-25
 Status: Needs-Revision
--- a/doc/spec/proposals/132-browser-check-tor-service.txt
+++ b/doc/spec/proposals/132-browser-check-tor-service.txt
@ -1,7 +1,5 @@
 Filename: 132-browser-check-tor-service.txt
 Title: A Tor Web Service For Verifying Correct Browser Configuration
-Version: $Revision$
-Last-Modified: $Date$
 Author: Robert Hogan
 Created: 2008-03-08
 Status: Draft
--- a/doc/spec/proposals/134-robust-voting.txt
+++ b/doc/spec/proposals/134-robust-voting.txt
@ -2,8 +2,10 @@ Filename: 134-robust-voting.txt
 Title: More robust consensus voting with diverse authority sets
 Author: Peter Palfrader
 Created: 2008-04-01
-Status: Accepted
-Target: 0.2.2.x
+Status: Rejected
+
+History:
+  2009 May 27: Added note on rejecting this proposal -- Nick

 Overview:

@ -103,3 +105,19 @@ Possible Attacks/Open Issues/Some thinking required:
 Q: Can this ever force us to build a consensus with authorities we do not
    recognize?
 A: No, we can never build a fully connected set with them in step 3.
+
+------------------------------
+
+I'm rejecting this proposal as insecure.
+
+Suppose that we have a clique of size N, and M hostile members in the
+clique.  If these hostile members stop declaring trust for up to M-1
+good members of the clique, the clique with the hostile members will
+in it will be larger than the one without them.
+
+The M hostile members will constitute a majority of this new clique
+when M > (N-(M-1)) / 2, or when M > (N + 1) / 3.  This breaks our
+requirement that an adversary must compromise a majority of authorities
+in order to control the consensus.
+
+-- Nick
--- a/doc/spec/proposals/135-private-tor-networks.txt
+++ b/doc/spec/proposals/135-private-tor-networks.txt
@ -1,7 +1,5 @@
 Filename: 135-private-tor-networks.txt
 Title: Simplify Configuration of Private Tor Networks
-Version: $Revision$
-Last-Modified: $Date$
 Author: Karsten Loesing
 Created: 29-Apr-2008
 Status: Closed
--- a/doc/spec/proposals/137-bootstrap-phases.txt
+++ b/doc/spec/proposals/137-bootstrap-phases.txt
@ -1,7 +1,5 @@
 Filename: 137-bootstrap-phases.txt
 Title: Keep controllers informed as Tor bootstraps
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 07-Jun-2008
 Status: Closed
--- a/doc/spec/proposals/138-remove-down-routers-from-consensus.txt
+++ b/doc/spec/proposals/138-remove-down-routers-from-consensus.txt
@ -1,7 +1,5 @@
 Filename: 138-remove-down-routers-from-consensus.txt
 Title: Remove routers that are not Running from consensus documents
-Version: $Revision$
-Last-Modified: $Date$
 Author: Peter Palfrader
 Created: 11-Jun-2008
 Status: Closed
--- a/doc/spec/proposals/140-consensus-diffs.txt
+++ b/doc/spec/proposals/140-consensus-diffs.txt
@ -1,12 +1,15 @@
 Filename: 140-consensus-diffs.txt
 Title: Provide diffs between consensuses
-Version: $Revision$
-Last-Modified: $Date$
 Author: Peter Palfrader
 Created: 13-Jun-2008
 Status: Accepted
 Target: 0.2.2.x

+0. History
+
+  22-May-2009: Restricted the ed format even more strictly for ease of
+  implementation. -nickm
+
 1. Overview.

  Tor clients and servers need a list of which relays are on the
@ -135,6 +138,10 @@ Target: 0.2.2.x
  Note that line numbers always apply to the file after all previous
  commands have already been applied.

+  The commands MUST apply to the file from back to front, such that
+  lines are only ever referred to by their position in the original
+  file.
+
  The "current line" is either the first line of the file, if this is
  the first command, the last line of a block we added in an append or
  change command, or the line immediate following a set of lines we just
--- a/doc/spec/proposals/141-jit-sd-downloads.txt
+++ b/doc/spec/proposals/141-jit-sd-downloads.txt
@ -1,7 +1,5 @@
 Filename: 141-jit-sd-downloads.txt
 Title: Download server descriptors on demand
-Version: $Revision$
-Last-Modified: $Date$
 Author: Peter Palfrader
 Created: 15-Jun-2008
 Status: Draft
@ -63,8 +61,8 @@ Status: Draft
  which tries to convey a server's capacity to clients.

  Currently we weigh servers differently for different purposes.  There
-  is a weigh for when we use a server as a guard node (our entry to the
-  Tor network), there is one weigh we assign servers for exit duties,
+  is a weight for when we use a server as a guard node (our entry to the
+  Tor network), there is one weight we assign servers for exit duties,
  and a third for when we need intermediate (middle) nodes.

 2.2 Exit information
@ -80,7 +78,7 @@ Status: Draft

 2.3 Capability information

-  Server descriptors contain information about the specific version or
+  Server descriptors contain information about the specific version of
  the Tor protocol they understand [proposal 105].

  Furthermore the server descriptor also contains the exact version of
--- a/doc/spec/proposals/142-combine-intro-and-rend-points.txt
+++ b/doc/spec/proposals/142-combine-intro-and-rend-points.txt
@ -1,7 +1,5 @@
 Filename: 142-combine-intro-and-rend-points.txt
 Title: Combine Introduction and Rendezvous Points
-Version: $Revision$
-Last-Modified: $Date$
 Author: Karsten Loesing, Christian Wilms
 Created: 27-Jun-2008
 Status: Dead
--- a/doc/spec/proposals/143-distributed-storage-improvements.txt
+++ b/doc/spec/proposals/143-distributed-storage-improvements.txt
@ -1,7 +1,5 @@
 Filename: 143-distributed-storage-improvements.txt
 Title: Improvements of Distributed Storage for Tor Hidden Service Descriptors
-Version: $Revision$
-Last-Modified: $Date$
 Author: Karsten Loesing
 Created: 28-Jun-2008
 Status: Open
--- a/doc/spec/proposals/145-newguard-flag.txt
+++ b/doc/spec/proposals/145-newguard-flag.txt
@ -1,7 +1,5 @@
 Filename: 145-newguard-flag.txt
 Title: Separate "suitable as a guard" from "suitable as a new guard"
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 1-Jul-2008
 Status: Open
--- a/doc/spec/proposals/146-long-term-stability.txt
+++ b/doc/spec/proposals/146-long-term-stability.txt
@ -1,7 +1,5 @@
 Filename: 146-long-term-stability.txt
 Title: Add new flag to reflect long-term stability
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 19-Jun-2008
 Status: Open
--- a/doc/spec/proposals/147-prevoting-opinions.txt
+++ b/doc/spec/proposals/147-prevoting-opinions.txt
@ -1,7 +1,5 @@
 Filename: 147-prevoting-opinions.txt
 Title: Eliminate the need for v2 directories in generating v3 directories
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 2-Jul-2008
 Status: Accepted
--- a/doc/spec/proposals/148-uniform-client-end-reason.txt
+++ b/doc/spec/proposals/148-uniform-client-end-reason.txt
@ -1,7 +1,5 @@
 Filename: 148-uniform-client-end-reason.txt
 Title: Stream end reasons from the client side should be uniform
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 2-Jul-2008
 Status: Closed
--- a/doc/spec/proposals/149-using-netinfo-data.txt
+++ b/doc/spec/proposals/149-using-netinfo-data.txt
@ -1,7 +1,5 @@
 Filename: 149-using-netinfo-data.txt
 Title: Using data from NETINFO cells
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 2-Jul-2008
 Status: Open
@ -24,14 +22,14 @@ Motivation
   idea of their own IP addresses, so they can publish correct
   descriptors.  This is also in NETINFO cells.

-Learning the time and IP
+Learning the time and IP address

   We need to think about attackers here.  Just because a router tells
   us that we have a given IP or a given clock skew doesn't mean that
   it's true.  We believe this information only if we've heard it from
   a majority of the routers we've connected to recently, including at
   least 3 routers.  Routers only believe this information if the
-   majority inclues at least one authority.
+   majority includes at least one authority.

 Avoiding MITM attacks

--- a/doc/spec/proposals/150-exclude-exit-nodes.txt
+++ b/doc/spec/proposals/150-exclude-exit-nodes.txt
@ -1,6 +1,5 @@
 Filename: 150-exclude-exit-nodes.txt
 Title: Exclude Exit Nodes from a circuit
-Version: $Revision$
 Author: Mfr
 Created: 2008-06-15
 Status: Closed
--- a/doc/spec/proposals/151-path-selection-improvements.txt
+++ b/doc/spec/proposals/151-path-selection-improvements.txt
@ -1,7 +1,5 @@
 Filename: 151-path-selection-improvements.txt
 Title: Improving Tor Path Selection
-Version:
-Last-Modified:
 Author: Fallon Chen, Mike Perry
 Created: 5-Jul-2008
 Status: Draft
--- a/doc/spec/proposals/152-single-hop-circuits.txt
+++ b/doc/spec/proposals/152-single-hop-circuits.txt
@ -1,7 +1,5 @@
 Filename: 152-single-hop-circuits.txt
 Title: Optionally allow exit from single-hop circuits 
-Version:
-Last-Modified:
 Author: Geoff Goodell
 Created: 13-Jul-2008
 Status: Closed
--- a/doc/spec/proposals/153-automatic-software-update-protocol.txt
+++ b/doc/spec/proposals/153-automatic-software-update-protocol.txt
@ -1,7 +1,5 @@
 Filename: 153-automatic-software-update-protocol.txt
 Title: Automatic software update protocol
-Version: $Revision$
-Last-Modified: $Date$
 Author: Jacob Appelbaum 
 Created: 14-July-2008
 Status: Superseded
--- a/doc/spec/proposals/154-automatic-updates.txt
+++ b/doc/spec/proposals/154-automatic-updates.txt
@ -1,7 +1,5 @@
 Filename: 154-automatic-updates.txt
 Title: Automatic Software Update Protocol
-Version: $Revision$
-Last-Modified: $Date$
 Author: Matt Edman
 Created: 30-July-2008
 Status: Superseded
--- a/doc/spec/proposals/155-four-hidden-service-improvements.txt
+++ b/doc/spec/proposals/155-four-hidden-service-improvements.txt
@ -1,7 +1,5 @@
 Filename: 155-four-hidden-service-improvements.txt
 Title: Four Improvements of Hidden Service Performance
-Version: $Revision$
-Last-Modified: $Date$
 Author: Karsten Loesing, Christian Wilms
 Created: 25-Sep-2008
 Status: Finished
--- a/doc/spec/proposals/156-tracking-blocked-ports.txt
+++ b/doc/spec/proposals/156-tracking-blocked-ports.txt
@ -1,7 +1,5 @@
 Filename: 156-tracking-blocked-ports.txt
 Title: Tracking blocked ports on the client side
-Version: $Revision$
-Last-Modified: $Date$
 Author: Robert Hogan
 Created: 14-Oct-2008
 Status: Open
--- a/doc/spec/proposals/157-specific-cert-download.txt
+++ b/doc/spec/proposals/157-specific-cert-download.txt
@ -1,7 +1,5 @@
 Filename: 157-specific-cert-download.txt
 Title: Make certificate downloads specific
-Version: $Revision$
-Last-Modified: $Date$
 Author: Nick Mathewson
 Created: 2-Dec-2008
 Status: Accepted
--- a/doc/spec/proposals/158-microdescriptors.txt
+++ b/doc/spec/proposals/158-microdescriptors.txt
@ -1,11 +1,20 @@
 Filename: 158-microdescriptors.txt
 Title: Clients download consensus + microdescriptors
-Version: $Revision$
-Last-Modified: $Date$
 Author: Roger Dingledine
 Created: 17-Jan-2009
 Status: Open

+0. History
+
+  15 May 2009: Substantially revised based on discussions on or-dev
+  from late January.  Removed the notion of voting on how to choose
+  microdescriptors; made it just a function of the consensus method.
+  (This lets us avoid the possibility of "desynchronization.")
+  Added suggestion to use a new consensus flavor.  Specified use of
+  SHA256 for new hashes. -nickm
+
+  15 June 2009: Cleaned up based on comments from Roger. -nickm
+
 1. Overview

  This proposal replaces section 3.2 of proposal 141, which was
@ -13,9 +22,7 @@ Status: Open
  circuit-building protocol to fetch a server descriptor inline at each
  circuit extend, we instead put all of the information that clients need
  either into the consensus itself, or into a new set of data about each
-  relay called a microdescriptor. The microdescriptor is a direct
-  transform from the relay descriptor, so relays don't even need to know
-  this is happening.
+  relay called a microdescriptor.

  Descriptor elements that are small and frequently changing should go
  in the consensus itself, and descriptor elements that are small and
@ -24,6 +31,10 @@ Status: Open
  them, we'll need to resume considering some design like the one in
  proposal 141.

+  Note also that any descriptor element which clients need to use to
+  decide which servers to fetch info about, or which servers to fetch
+  info from, needs to stay in the consensus.
+
 2. Motivation

  See
@ -36,99 +47,91 @@ Status: Open
 3. Design

  There are three pieces to the proposal. First, authorities will list in
-  their votes (and thus in the consensus) what relay descriptor elements
-  are included in the microdescriptor, and also list the expected hash
-  of microdescriptor for each relay. Second, directory mirrors will serve
-  microdescriptors. Third, clients will ask for them and cache them.
+  their votes (and thus in the consensus) the expected hash of
+  microdescriptor for each relay. Second, authorities will serve
+  microdescriptors, directory mirrors will cache and serve
+  them. Third, clients will ask for them and cache them.

 3.1. Consensus changes

-  V3 votes should include a new line:
-    microdescriptor-elements bar baz foo
-  listing each descriptor element (sorted alphabetically) that authority
-  included when it calculated its expected microdescriptor hashes.
+  If the authorities choose a consensus method of a given version or
+  later, a microdescriptor format is implicit in that version.
+  A microdescriptor should in every case be a pure function of the
+  router descriptor and the consensus method.

-  We also need to include the hash of each expected microdescriptor in
-  the routerstatus section. I suggest a new "m" line for each stanza,
-  with the base64 of the hash of the elements that the authority voted
-  for above.
+  In votes, we need to include the hash of each expected microdescriptor
+  in the routerstatus section. I suggest a new "m" line for each stanza,
+  with the base64 of the SHA256 hash of the router's microdescriptor.
+
+  For every consensus method that an authority supports, it includes a
+  separate "m" line in each router section of its vote, containing:
+    "m" SP methods 1*(SP AlgorithmName "=" digest) NL
+  where methods is a comma-separated list of the consensus methods
+  that the authority believes will produce "digest".
+
+  (As with base64 encoding of SHA1 hashes in consensuses, let's
+  omit the trailing =s)

  The consensus microdescriptor-elements and "m" lines are then computed
  as described in Section 3.1.2 below.

-  I believe that means we need a new consensus-method "6" that knows
-  how to compute the microdescriptor-elements and add "m" lines.
+  (This means we need a new consensus-method that knows
+  how to compute the microdescriptor-elements and add "m" lines.)
+
+  The microdescriptor consensus uses the directory-signature format from
+  proposal 162, with the "sha256" algorithm.
+

 3.1.1. Descriptor elements to include for now

-  To start, the element list that authorities suggest should be
-    family onion-key
-
-  (Note that the or-dev posts above only mention onion-key, but if
-  we don't also include family then clients will never learn it. It
-  seemed like it should be relatively static, so putting it in the
-  microdescriptor is smarter than trying to fit it into the consensus.)
-
-  We could imagine a config option "family,onion-key" so authorities
-  could change their voted preferences without needing to upgrade.
+  In the first version, the microdescriptor should contain the
+  onion-key element, and the family element from the router descriptor,
+  and the exit policy summary as currently specified in dir-spec.txt.

 3.1.2. Computing consensus for microdescriptor-elements and "m" lines

-  One approach is for the consensus microdescriptor-elements line to
-  include every element listed by a majority of authorities, sorted. The
-  problem here is that it will no longer be deterministic what the correct
-  hash for the "m" line should be. We could imagine telling the authority
-  to go look in its descriptor and produce the right hash itself, but
-  we don't want consensus calculation to be based on external data like
-  that. (Plus, the authority may not have the descriptor that everybody
-  else voted to use.)
+  When we are generating a consensus, we use whichever m line
+  unambiguously corresponds to the descriptor digest that will be
+  included in the consensus.

-  The better approach is to take the exact set that has the most votes
-  (breaking ties by the set that has the most elements, and breaking
-  ties after that by whichever is alphabetically first). That will
-  increase the odds that we actually get a microdescriptor hash that
-  is both a) for the descriptor we're putting in the consensus, and b)
-  over the elements that we're declaring it should be for.
+  (If different votes have different microdescriptor digests for a
+  single <descriptor-digest, consensus-method> pair, then at least one
+  of the authorities is broken.  If this happens, the consensus should
+  contain whichever microdescriptor digest is most common.  If there is
+  no winner, we break ties in the favor of the lexically earliest.
+  Either way, we should log a warning: there is definitely a bug.)

-  Then the "m" line for a given relay is the one that gets the most votes
-  from authorities that both a) voted for the microdescriptor-elements
-  line we're using, and b) voted for the descriptor we're using.
+  The "m" lines in a consensus contain only the digest, not a list of
+  consensus methods.

-  (If there's a tie, use the smaller hash. But really, if there are
-  multiple such votes and they differ about a microdescriptor, we caught
-  one of them lying or being buggy. We should log it to track down why.)
+3.1.3. A new flavor of consensus

-  If there are no such votes, then we leave out the "m" line for that
-  relay. That means clients should avoid it for this time period. (As
-  an extension it could instead mean that clients should fetch the
-  descriptor and figure out its microdescriptor themselves. But let's
-  not get ahead of ourselves.)
+  Rather than inserting "m" lines in the current consensus format,
+  they should be included in a new consensus flavor (see proposal
+  162).

-  It would be nice to have a more foolproof way to agree on what
-  microdescriptor hash each authority should vote for, so we can avoid
-  missing "m" lines. Just switching to a new consensus-method each time
-  we change the set of microdescriptor-elements won't help though, since
-  each authority will still have to decide what hash to vote for before
-  knowing what consensus-method will be used.
+  This flavor can safely omit descriptor digests.

-  Here's one way we could do it. Each vote / consensus includes
-  the microdescriptor-elements that were used to compute the hashes,
-  and also a preferred-microdescriptor-elements set. If an authority
-  has a consensus from the previous period, then it should use the
-  consensus preferred-microdescriptor-elements when computing its votes
-  for microdescriptor-elements and the appropriate hashes in the upcoming
-  period. (If it has no previous consensus, then it just writes its
-  own preferences in both lines.)
+  When we implement this voting method, we can remove the exit policy
+  summary from the current "ns" flavor of consensus, since no current
+  clients use them, and they take up about 5% of the compressed
+  consensus.

-3.2. Directory mirrors serve microdescriptors
+  This new consensus flavor should be signed with the sha256 signature
+  format as documented in proposal 162.

-  Directory mirrors should then read the microdescriptor-elements line
-  from the consensus, and learn how to answer requests. (Directory mirrors
-  continue to serve normal relay descriptors too, a) to serve old clients
-  and b) to be able to construct microdescriptors on the fly.)
+3.2. Directory mirrors fetch, cache, and serve microdescriptors

-  The microdescriptors with hashes <D1>,<D2>,<D3> should be available at:
-    http://<hostname>/tor/micro/d/<D1>+<D2>+<D3>.z
+  Directory mirrors should fetch, catch, and serve each microdescriptor
+  from the authorities.  (They need to continue to serve normal relay
+  descriptors too, to handle old clients.)
+
+  The microdescriptors with base64 hashes <D1>,<D2>,<D3> should be
+  available at:
+    http://<hostname>/tor/micro/d/<D1>-<D2>-<D3>.z
+  (We use base64 for size and for consistency with the consensus
+  format. We use -s instead of +s to separate these items, since
+  the + character is used in base64 encoding.)

  All the microdescriptors from the current consensus should also be
  available at:
@ -136,24 +139,9 @@ Status: Open
  so a client that's bootstrapping doesn't need to send a 70KB URL just
  to name every microdescriptor it's looking for.

-  The format of a microdescriptor is the header line
-  "microdescriptor-header"
-  followed by each element (keyword and body), alphabetically. There's
-  no need to mention what hash it's for, since it's self-identifying:
-  you can hash the elements to learn this.
-
-  (Do we need a footer line to show that it's over, or is the next
-  microdescriptor line or EOF enough of a hint? A footer line wouldn't
-  hurt much. Also, no fair voting for the microdescriptor-element
-  "microdescriptor-header".)
-
+  Microdescriptors have no header or footer.
  The hash of the microdescriptor is simply the hash of the concatenated
-  elements -- not counting the header line or hypothetical footer line.
-  Unless you prefer that?
-
-  Is there a reasonable way to version these things? We could say that
-  the microdescriptor-header line can contain arguments which clients
-  must ignore if they don't understand them. Any better ways?
+  elements.

  Directory mirrors should check to make sure that the microdescriptors
  they're about to serve match the right hashes (either the hashes from
@ -170,10 +158,14 @@ Status: Open
  When a client gets a new consensus, it looks to see if there are any
  microdescriptors it needs to learn. If it needs to learn more than
  some threshold of the microdescriptors (half?), it requests 'all',
-  else it requests only the missing ones.
+  else it requests only the missing ones.  Clients MAY try to
+  determine whether the upload bandwidth for listing the
+  microdescriptors they want is more or less than the download
+  bandwidth for the microdescriptors they do not want.

  Clients maintain a cache of microdescriptors along with metadata like
-  when it was last referenced by a consensus. They keep a microdescriptor
+  when it was last referenced by a consensus, and which identity key
+  it corresponds to.  They keep a microdescriptor
  until it hasn't been mentioned in any consensus for a week. Future
  clients might cache them for longer or shorter times.

@ -190,18 +182,17 @@ Status: Open
  Another future option would be to fetch some of the microdescriptors
  anonymously (via a Tor circuit).

+  Another crazy option (Roger's phrasing) is to do decoy fetches as
+  well.
+
 4. Transition and deployment

  Phase one, the directory authorities should start voting on
-  microdescriptors and microdescriptor elements, and putting them in the
-  consensus. This should happen during the 0.2.1.x series, and should
-  be relatively easy to do.
+  microdescriptors, and putting them in the consensus.

  Phase two, directory mirrors should learn how to serve them, and learn
-  how to read the consensus to find out what they should be serving. This
-  phase could be done either in 0.2.1.x or early in 0.2.2.x, depending
-  on how messy it turns out to be and how quickly we get around to it.
+  how to read the consensus to find out what they should be serving.

  Phase three, clients should start fetching and caching them instead
-  of normal descriptors. This should happen post 0.2.1.x.
+  of normal descriptors.

--- a/doc/spec/proposals/159-exit-scanning.txt
+++ b/doc/spec/proposals/159-exit-scanning.txt
@ -1,7 +1,5 @@
 Filename: 159-exit-scanning.txt
 Title: Exit Scanning
-Version: $Revision$
-Last-Modified: $Date$
 Author: Mike Perry
 Created: 13-Feb-2009
 Status: Open
--- a/doc/spec/proposals/160-bandwidth-offset.txt
+++ b/doc/spec/proposals/160-bandwidth-offset.txt
@ -0,0 +1,105 @@
+Filename: 160-bandwidth-offset.txt
+Title: Authorities vote for bandwidth offsets in consensus
+Author: Roger Dingledine
+Created: 4-May-2009
+Status: Open
+Target: 0.2.2.x
+
+1. Motivation
+
+  As part of proposal 141, we moved the bandwidth value for each relay
+  into the consensus. Now clients can know how they should load balance
+  even before they've fetched the corresponding relay descriptors.
+
+  Putting the bandwidth in the consensus also lets the directory
+  authorities choose more accurate numbers to advertise, if we come up
+  with a better algorithm for deciding weightings.
+
+  Our original plan was to teach directory authorities how to measure
+  bandwidth themselves; then every authority would vote for the bandwidth
+  it prefers, and we'd take the median of votes as usual.
+
+  The problem comes when we have 7 authorities, and only a few of them
+  have smarter bandwidth allocation algorithms. So long as the majority
+  of them are voting for the number in the relay descriptor, the minority
+  that have better numbers will be ignored.
+
+2. Options
+
+  One fix would be to demand that every authority also run the
+  new bandwidth measurement algorithms: in that case, part of the
+  responsibility of being an authority operator is that you need to run
+  this code too. But in practice we can't really require all current
+  authority operators to do that; and if we want to expand the set of
+  authority operators even further, it will become even more impractical.
+  Also, bandwidth testing adds load to the network, so we don't really
+  want to require that the number of concurrent bandwidth tests match
+  the number of authorities we have.
+
+  The better fix is to allow certain authorities to specify that they are
+  voting on bandwidth measurements: more accurate bandwidth values that
+  have actually been evaluated. In this way, authorities can vote on 
+  the median measured value if sufficient measured votes exist for a router,
+  and otherwise fall back to the median value taken from the published router
+  descriptors.
+
+3. Security implications
+
+  If only some authorities choose to vote on an offset, then a majority of
+  those voting authorities can arbitrarily change the bandwidth weighting
+  for the relay. At the extreme, if there's only one offset-voting
+  authority, then that authority can dictate which relays clients will
+  find attractive.
+
+  This problem isn't entirely new: we already have the worry wrt
+  the subset of authorities that vote for BadExit.
+
+  To make it not so bad, we should deploy at least three offset-voting
+  authorities.
+
+  Also, authorities that know how to vote for offsets should vote for
+  an offset of zero for new nodes, rather than choosing not to vote on
+  any offset in those cases.
+
+4. Design
+
+  First, we need a new consensus method to support this new calculation.
+
+  Now v3 votes can have an additional value on the "w" line:
+    "w Bandwidth=X Measured=" INT.
+
+  Once we're using the new consensus method, the new way to compute the
+  Bandwidth weight is by checking if there are at least 3 "Measured"
+  votes. If so, the median of these is taken. Otherwise, the median
+  of the "Bandwidth=" values are taken, as described in Proposal 141.
+
+  Then the actual consensus looks just the same as it did before,
+  so clients never have to know that this additional calculation is
+  happening.
+
+5. Implementation
+
+  The Measured values will be read from a file provided by the scanners
+  described in proposal 161. Files with a timestamp older than 3 days
+  will be ignored.
+
+  The file will be read in from dirserv_generate_networkstatus_vote_obj()
+  in a location specified by a new config option "V3MeasuredBandwidths".
+  A helper function will be called to populate new 'measured' and
+  'has_measured' fields of the routerstatus_t 'routerstatuses' list with 
+  values read from this file.
+
+  An additional for_vote flag will be passed to 
+  routerstatus_format_entry() from format_networkstatus_vote(), which will 
+  indicate that the "Measured=" string should be appended to the "w Bandwith=" 
+  line with the measured value in the struct.
+
+  routerstatus_parse_entry_from_string() will be modified to parse the
+  "Measured=" lines into routerstatus_t struct fields.
+
+  Finally, networkstatus_compute_consensus() will set rs_out.bandwidth 
+  to the median of the measured values if there are more than 3, otherwise
+  it will use the bandwidth value median as normal.
+
+
+
--- a/doc/spec/proposals/161-computing-bandwidth-adjustments.txt
+++ b/doc/spec/proposals/161-computing-bandwidth-adjustments.txt
@ -0,0 +1,174 @@
+Title: Computing Bandwidth Adjustments
+Filename: 161-computing-bandwidth-adjustments.txt
+Author: Mike Perry
+Created: 12-May-2009
+Target: 0.2.2.x
+Status: Open
+
+
+1. Motivation
+
+  There is high variance in the performance of the Tor network. Despite
+  our efforts to balance load evenly across the Tor nodes, some nodes are
+  significantly slower and more overloaded than others.
+
+  Proposal 160 describes how we can augment the directory authorities to
+  vote on measured bandwidths for routers. This proposal describes what
+  goes into the measuring process.
+
+
+2. Measurement Selection
+
+  The general idea is to determine a load factor representing the ratio
+  of the capacity of measured nodes to the rest of the network. This load
+  factor could be computed from three potentially relevant statistics:
+  circuit failure rates, circuit extend times, or stream capacity.
+
+  Circuit failure rates and circuit extend times appear to be
+  non-linearly proportional to node load. We've observed that the same
+  nodes when scanned at US nighttime hours (when load is presumably
+  lower) exhibit almost no circuit failure, and significantly faster
+  extend times than when scanned during the day.
+
+  Stream capacity, however, is much more uniform, even during US
+  nighttime hours. Moreover, it is a more intuitive representation of
+  node capacity, and also less dependent upon distance and latency
+  if amortized over large stream fetches.
+
+
+3. Average Stream Bandwidth Calculation
+
+  The average stream bandwidths are obtained by dividing the network into
+  slices of 50 nodes each, grouped according to advertised node bandwidth.
+
+  Two hop circuits are built using nodes from the same slice, and a large
+  file is downloaded via these circuits. The file sizes are set based
+  on node percentile rank as follows:
+    
+     0-10: 2M
+     10-20: 1M
+     20-30: 512k
+     30-50: 256k
+     50-100: 128k
+
+  These sizes are based on measurements performed during test scans.
+
+  This process is repeated until each node has been chosen to participate
+  in at least 5 circuits.
+
+
+4. Ratio Calculation
+
+  The ratios are calculated by dividing each measured value by the 
+  network-wide average.
+
+
+5. Ratio Filtering
+
+  After the base ratios are calculated, a second pass is performed
+  to remove any streams with nodes of ratios less than X=0.5 from
+  the results of other nodes. In addition, all outlying streams
+  with capacity of one standard deviation below a node's average
+  are also removed.
+
+  The final ratio result will be greater of the unfiltered ratio
+  and the filtered ratio.
+
+
+6. Pseudocode for Ratio Calculation Algorithm
+
+  Here is the complete pseudocode for the ratio algorithm:
+
+    Slices = {S | S is 50 nodes of similar consensus capacity}
+    for S in Slices:
+      while exists node N in S with circ_chosen(N) < 7:
+        fetch_slice_file(build_2hop_circuit(N, (exit in S)))
+      for N in S:
+        BW_measured(N) = MEAN(b | b is bandwidth of a stream through N)
+        Bw_stddev(N) = STDDEV(b | b is bandwidth of a stream through N)
+      Bw_avg(S) = MEAN(b | b = BW_measured(N) for all N in S)  
+      for N in S:
+        Normal_Streams(N) = {stream via N | bandwidth >= BW_measured(N)} 
+        BW_Norm_measured(N) =  MEAN(b | b is a bandwidth of Normal_Streams(N))
+
+    Bw_net_avg(Slices) = MEAN(BW_measured(N) for all N in Slices)
+    Bw_Norm_net_avg(Slices) = MEAN(BW_Norm_measured(N) for all N in Slices)
+
+    for N in all Slices:
+      Bw_net_ratio(N) = Bw_measured(N)/Bw_net_avg(Slices)
+      Bw_Norm_net_ratio(N) = BW_Norm_measured(N)/Bw_Norm_net_avg(Slices)
+
+      ResultRatio(N) = MAX(Bw_net_ratio(N), Bw_Norm_net_ratio(N))
+
+
+7. Security implications
+
+  The ratio filtering will deal with cases of sabotage by dropping
+  both very slow outliers in stream average calculations, as well
+  as dropping streams that used very slow nodes from the calculation
+  of other nodes.
+
+  This scheme will not address nodes that try to game the system by
+  providing better service to scanners. The scanners can be detected
+  at the entry by IP address, and at the exit by the destination fetch
+  IP.
+
+  Measures can be taken to obfuscate and separate the scanners' source
+  IP address from the directory authority IP address. For instance,
+  scans can happen offsite and the results can be rsynced into the
+  authorities. The destination server IP can also change.
+ 
+  Neither of these methods are foolproof, but such nodes can already
+  lie about their bandwidth to attract more traffic, so this solution
+  does not set us back any in that regard.
+
+
+8. Parallelization
+
+  Because each slice takes as long as 6 hours to complete, we will want
+  to parallelize as much as possible. This will be done by concurrently
+  running multiple scanners from each authority to deal with different
+  segments of the network. Each scanner piece will continually loop 
+  over a portion of the network, outputting files of the form:
+
+   node_id=<idhex> SP strm_bw=<BW_measured(N)> SP 
+         filt_bw=<BW_Norm_measured(N)> ns_bw=<CurrentConsensusBw(N)> NL
+
+  The most recent file from each scanner will be periodically gathered 
+  by another script that uses them to produce network-wide averages 
+  and calculate ratios as per the algorithm in section 6. Because nodes 
+  may shift in capacity, they may appear in more than one slice and/or 
+  appear more than once in the file set. The most recently measured
+  line will be chosen in this case.
+
+
+9. Integration with Proposal 160
+
+  The final results will be produced for the voting mechanism
+  described in Proposal 160 by multiplying the derived ratio by
+  the average published consensus bandwidth during the course of the
+  scan, and taking the weighted average with the previous consensus
+  bandwidth:
+
+     Bw_new = Round((Bw_current * Alpha + Bw_scan_avg*Bw_ratio)/(Alpha + 1))
+
+  The Alpha parameter is a smoothing parameter intended to prevent
+  rapid oscillation between loaded and unloaded conditions. It is
+  currently fixed at 0.333.
+
+  The Round() step consists of rounding to the 3 most significant figures
+  in base10, and then rounding that result to the nearest 1000, with 
+  a minimum value of 1000.
+
+  This will produce a new bandwidth value that will be output into a 
+  file consisting of lines of the form:
+
+     node_id=<idhex> SP bw=<Bw_new> NL
+ 
+  The first line of the file will contain a timestamp in UNIX time()
+  seconds. This will be used by the authority to decide if the 
+  measured values are too old to use.
+ 
+  This file can be either copied or rsynced into a directory readable
+  by the directory authority.
+
--- a/Show More
+++ b/Show More