codeberg.org/japh/psv

PSV - Pipe Separated Values

(or "Data Table Parser And Generator")

Introduction

psv was initially created to help write "nice looking tables for gherkin scenarios". Don't worry if you don't know what "gherkin scenarios" means, the important bits are "nice looking" and "tables [of data in a text file]"

Now, psv is a unix command line utility and go package to help maintain and utilise data in simple, text-based tables using a variant of the Pipe Separated Values format.

In short, psv helps you "draw" tables of data in text files and/or use tabular data programmatically.

Index

• Introductory Examples
• Detailed Description
• Main Features
• Documentation Links
• Installation
• Alternatives
• Copyright

Introductory examples

Creating psv Tables Textually

For example, psv can help you turn this, deliberately sloppily hacked up text (for demonstration purposes):

                  Controls
    +--+
        |key|action             |                alternative
    | - | --- |
|h|left
                      |   j|down
|k  |up
        |  l |      right ||||||||
    :  :
    |   :wq |   write & quit |      ZZ
        +----------

into this:

                  Controls
    +-----+--------------+-------------+
    | key | action       | alternative |
    | --- | ------------ | ----------- |
    | h   | left         |             |
    | j   | down         |             |
    | k   | up           |             |
    | l   | right        |             |
    :     :              :             :
    | :wq | write & quit | ZZ          |
    +----------------------------------+

with a single call to psv (in this case, the vim ¹ command: vip!psv ²).

The magic being that each line beginning with a | is split into cells and re-formatted so they all get the same level of indentation, that all columns line up nicely and that any superfluous or missing |'s are removed or added as needed. Additionally, the : : : line is also considered to be part of the table and is aligned appropriately. (see ruler formatting)

Using psv Tables Programmatically

psv Tables can also help improve the readibility of test data.

Here is an example of an actual test suite (containing 14 individual unit tests) from psv's own unit testing code (sort_test.go):

func TestSingleSectionSorting(t *testing.T) {

    testTable, _ := psv.TableFromString(`
        | 0 | b | 3  | partial
        | 1 | D
        | 2 | E | 5
        | 3 | a | 4  | unequal
        | 4 | c | 20
        | 5 | C | 10 | row | lengths
        | 6 | e | 5
        | 7 | d | 7
        `)

    testCases := sortingTestCasesFromTable(`
	| name                         | sort  | columns | exp-col | exp-rows        |
	| ---------------------------- | ----- | ------- | ------- | --------------- |
	| no sort                      | false |         |         | 0 1 2 3 4 5 6 7 |
	| default sort                 |       |         |         | 0 1 2 3 4 5 6 7 |
	| sort only when asked to      | false | 2       |         | 0 1 2 3 4 5 6 7 |
	| reverse default sort         |       | ~       |         | 7 6 5 4 3 2 1 0 |
	| reverse reverse default sort |       | ~~      |         | 0 1 2 3 4 5 6 7 |
	| indexed column sort          |       | 2       |         | 3 0 4 5 7 1 6 2 |
	| indexed column sort          |       | 2       | 2       | a b c C d D e E |
	| reverse column sort          |       | ~2      |         | 2 6 1 7 5 4 0 3 |
	| third column sort            |       | 3       |         | 1 5 4 0 3 2 6 7 |
	| numeric sort                 |       | #3      |         | 1 0 3 2 6 7 5 4 |
	| reverse numeric sort         |       | ~#3     |         | 4 5 7 6 2 3 0 1 |
	| numeric reverse sort         |       | #~3     |         | 4 5 7 6 2 3 0 1 |
	| reverse reverse column sort  |       | ~ #~3   |         | 1 0 3 2 6 7 5 4 |
	| partial column sort          |       | 4 2     |         | 4 7 1 6 2 0 5 3 |
	| non-existent column sort     |       | 9       |         | 0 1 2 3 4 5 6 7 |
	`)

    runSortingTestCases(t, testTable, testCases)
}

In the example above, two tables are defined:

• testTable is the reference table to be tested

  • it simply contains a few rows of pseudo-random data, in various forms suitable for testing some features of psv

• testCases then defines a series of individual unit tests to be run on testTable

  • the first two rows (|name|... and |---|...) define a header for the table
    • psv infers(!) that "a single, separate row, followed by a ruler" means that the text in each cell defines the name of each respective column, for all following rows, or until another header is found.
    • this is only used for convenience and is not required in any way!
      • but it allows us to discard or rearrange columns without regard for how the table data is converted to sortingTestCase structs
      • see the sortingTestCasesFromTable function in the source code for details
  • each row after the ruler then defines a single unit-test to be run against the testTable
  • to add a new unit test, you just need to add a row to the testCases table
    • you can of course use psv to reformat the table if your column alignment gets messed up :smile:
  • the sortingTestCasesFromTable() function converts the table of strings into a slice of sortingTestCase structs

• finally, the runSortingTestCases() function sorts the testTable according to the conditions in each sortingTestCase and checks that the results match the expectations.

Detailed Description

psv reads, formats and writes simple tables of data in text files.

In doing so, psv focuses on human readibility and ease of use, rather than trying to provide a loss-less, ubiquitous, machine-readable data transfer format.

The same could be said of markdown, and indeed, psv can be used to generate github-style markdown tables that look nice in their markdown source code, and not just after they have been converted to HTML by the markdown renderer.

Another intended use case is data tables in Gherkin files, which are a central component of Behaviour Driven Development (BDD).

However, the real reason for creating psv was to be able to use text tables as the source of data for running automated tests. Hence the go package.

Main Features

• normalisation of rows and columns, so that every row has the same number of cells
• automatic table indentation and column alignment
• the ability to automatically draw horizontal separation lines, called rulers
• the ability to re-format existing tables, while leaving lines which "do not look like table rows" unchanged
• a simple way to read data from tables into go programs via the psv go package
• the (limited) ability to sort table data
  • without interfering with the rest of the table's formatting
• and more ...

Not Supported

psv is not intended to replace spreadsheets etc :smile:

Among a myriad of other non-features, the following are definitely not supported by psv:

• the inclusion of | characters in a cell's data
• multi-line cell data
• any kind of cell merging or splitting
• sorting of complex data formats, including:
  • date and/or timestamps (unless they are in ISO-8601 format, which sorts nicely)
  • signed numbers (+ and - signs confuse go's collators :frowning:)
  • floating point numbers
  • scientific notation
  • hexadecimal notation
• ...

Design Principles

• self contained
  • psv is a single go binary with no external dependencies
  • the psv go package is a single package, also with no external dependecies other than go's standard packages
    • exception: I do include another package of mine to provide simplified testing with meaningful success and error messages.
  • all psv actions occur locally (no network access required)
• non-destructive
  • if psv doesn't know how to interperet a line of text, the text remains unchanged
    • only data rows (lines beginning with a |) and rulers are re-formatted, all other lines remain unchanged
• idempotent
  • any table generated by psv can also be read be psv
  • running a formatted table through psv again must not change the table in any way
• easy of use
  • normal use should not require any configuration or additional parameters

TODO's

• add ability to configure the scanner

  • allow auto-indent detection
    • -I detect indent by capturing the indent before the first | encountered
  • explicitly specify ruler characters (for cli)
    • default autodetect
    • explicit rulers
      • turns off autodetection
      • allows the use of + and - as data
      • options:
        • -rh '-' horizontal ruler
        • -ro '|' outer ruler
        • -ri ':' inner ruler
        • -rc '+' corners
        • -rp 'ophi'
          • o outer vertical ruler
          • p padding character
          • h horizontal ruler (default: same as padding character)
          • i inner vertical ruler (default: same as outer ruler)

• Replace table.Data with table.DataRows

Documentation Links

• Table Formatting Rules
• Using the psv command
• Using the psv go package
• Some Examples

Installation

psv consists of two components: the psv command and the psv go package.

To use the psv command, you only need the psv binary in your PATH, e.g. ~/bin/psv (see binary installation below).

If you don't want to install "a binary, downloaded from the 'net", you can download the source, (inspect it :smile:), and build your own version.

Source Installation

Prerequisites

• go 1.18 or later
• make (optional, but recommended)

Build Steps

Clone the psv git repository and use make to build, test and install psv in your $GOBIN directory (typically $GOPATH/bin or ~/Go/bin)

git clone -o codeberg https://codeberg.org/japh/psv
cd psv
make install
psv -v

Binary Installation

Note: currently only available for darwin amd64 (64-bit Intel Macs)

• download the latest psv.gz from https://codeberg.org/japh/psv/releases
• verify psv.gz with gpg --verify psv.gz.asc
• compare psv.gz's checksums against those provided with shasum -c psv.gz.sha256
• unpack psv.gz with gunzip psv.gz
• copy psv to any directory in your $PATH, or use it directly via ./psv
• don't forget to check that it is executable, e.g. chmod +x psv

Now you can use the psv command...

Using The psv Package In Go Projects

Prerequisites

• go 1.18 or later

To use psv in your go project, simply import codeberg.org/japh/psv and go mod tidy will download it, build it and make it available for your project.

See the psv package documentation for the API and code examples.

Alternatives

• csv, tsv and delimeter-separated-values tables | wikipedia

  • generally, psv tables are just a single type of delimeter separated values format

• ASCII Table Writer

  • go package for creating tables of almost any form
  • more traditional table.SetHeader, table.SetFooter() interface
  • more features (incl. colors)
  • does not read tables
    • no good for defining test cases etc in code

• psv-spec

  • an attempt to standardize a CSV replacement using pipes as the delimiter
  • focuses on electronic data transfers
  • does not provide a tabular layout
  • escaping just |, \, \n and \r is nice
    • but does not allow for whitespace quoting
    • future: | " " | could be used by psv to represent a space

Copyright

Copyright 2022 Stephen Riehm [e-mail]

Footnotes

1. You don't have to use vim! psv can be used from any editor that lets you pipe text through shell commands. ↩

2. which translates to:

   • v start a visual selection ...
   • i select everything in ...
   • p the current paragraph
   • !psv and replace the current selection with whatever psv makes of it.
   ↩