readorder

== Readorder

• Homepage[http://copiousfreetime.rubyforge.org/readorder/]
• {Rubyforge Project}[http://rubyforge.org/projects/copiousfreetime/]
• email jeremy at copiousfreetime dot org
• git clone git://github.com/copiousfreetime/readorder.git

== DESCRIPTION

Readorder orders a list of files into a more effective read order.

You would possibly want to use readorder in a case where you know ahead of time that you have a large quantity of files on disc to process. You can give that list off those files and it will report back to you the order in which you should process them to make most effective use of your disc I/O.

Given a list of filenames, either on the command line or via stdin, readorder will output the filenames in an order that should increase the I/O throughput when the files corresponding to the filenames are read off of disc.

The output order of the filenames can either be in inode order or physical disc block order. This is dependent upon operating system support and permission level of the user running readorder.

== COMMANDS

=== Sort

Given a list of filenames, either on the command line or via stdin, output the filenames in an order that should increase the I/O throughput when the contents files are read from disc.

==== Synopsis

readorder sort [filelist*] [options]+

filelist (-1 ~> filelist=#IO:0x1277e4) The files containing filenames --inode Only use inode order do not attempt physical block order --log-level=log-level (0 ~> log-level=info) The verbosity of logging, one of [ debug, info, warn, error, fatal ] --log-file=log-file (0 ~> log-file) Log to this file instead of stderr --output=output (0 ~> output) Where to write the output --error-filelist=error-filelist (0 ~> error-filelist) Write all the files from the filelist that had errors to this file --help, -h

==== Example Output

=== Analyze

Take the list of filenames and output an analysis of the volume of data in those files.

==== Synopsis

readorder analyze [filelist*] [options]+

filelist (-1 ~> filelist=#IO:0x1277e4) The files containing filenames --log-level=log-level (0 ~> log-level=info) The verbosity of logging, one of [ debug, info, warn, error, fatal ] --log-file=log-file (0 ~> log-file) Log to this file instead of stderr --output=output (0 ~> output) Where to write the output --error-filelist=error-filelist (0 ~> error-filelist) Write all the files from the filelist that had errors to this file --data-csv=data-csv (0 ~> data-csv) Write the raw data collected to this csv file --help, -h

==== Example Output

=== Test

Give a list of filenames, either on the commandline or via stdin, take a random subsample of them and read all the contents of those files in different orders.

• in initial given order
• in inode order
• in physical block order

Output a report of the various times take to read the files.

This command requires elevated priveleges to run. It will purge your disc cache multiple times while running, and will spike the I/O of your machine. Run with care.

==== Synopsis

readorder test [filelist*] [options]+

filelist (-1 ~> filelist=#IO:0x1277e4) The files containing filenames --percentage=percentage (0 ~> int(percentage)) What random percentage of input files to select --log-level=log-level (0 ~> log-level=info) The verbosity of logging, one of [ debug, info, warn, error, fatal ] --log-file=log-file (0 ~> log-file) Log to this file instead of stderr --error-filelist=error-filelist (0 ~> error-filelist) Write all the files from the filelist that had errors to this file --help, -h

==== Example result

                          Test Using First Of                           
========================================================================

  Total files read :         8052
  Total bytes read :      6575824
  Minimum filesize :          637
  Average filesize :          816.670
  Maximum filesize :         1393
  Stddev of sizes  :           86.936

                  read order   Elapsed time (sec)  Read rate (bytes/sec)
------------------------------------------------------------------------
              original_order              352.403              18659.944
                inode_number               53.606             122669.175
 first_physical_block_number               47.520             138379.024

This is the output of a a readorder test command run on a directory on a ReiserFS filesytem containing 805,038 files, constituting 657,543,700 bytes of data. A sample of 1% of the files was used for the test.

If we process them in their original order we can see that this will potentially take us 9.78 hours. If we process them in physical block number order that is reduces to 1.31 hours.

== CREDITS

• Linux System Programming by Robert Love
• {readahead project}[https://fedorahosted.org/readahead/]

== ISC LICENSE

Copyright (c) 2009, Jeremy Hinegardner

Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.