ffe — Flat File Extractor

ffe is a versatile command-line tool for extracting, parsing, filtering, transforming, and anonymizing fields from text flat files (fixed-length or delimited/CSV) and binary files.

Input formats and arbitrary output formatting are defined using an intuitive configuration file. ffe can read standard input and write to standard output, making it an ideal component for UNIX pipelines, ETL jobs, and data migration workflows.

---

Table of Contents

• Features
• Installation
  • Prerequisites
  • Building from Source
• Quick Start Tutorial
  • 1. Converting Fixed-Length Records to XML
  • 2. Parsing Delimited Files (CSV / TSV)
  • 3. Filtering Records with Expressions
  • 4. Data Anonymization
• Command-Line Options
• Configuration Overview
  • Structures
  • Records
  • Outputs
  • Output Format Directives
  • Anonymization
  • Lookup Tables and Filters
• Documentation & Future Conversions
  • Reading the Manual
  • How to Convert doc/ffe.texi to Markdown
• License & Author

---

Features

• Multiple Input Formats:
  • Fixed-length: Flat files where fields occupy constant byte counts.
  • Separated / Delimited: CSV, TSV, or arbitrary character-separated files (handles multi-character separators and quotation rules).
  • Binary: Native binary structs, fixed and variable-length, with full endianness awareness (little, big, or system endian).
• Multi-Record Support: A single file can contain several distinct record types (e.g., headers, body items, and trailers) identified by positions, strings, or regular expressions (id / rid).
• Flexible Output Formatting: Output is completely independent of input structure. Produce XML, JSON, SQL statements, custom CSV, fixed-length reports, or human-readable tables using printf-style directives.
• Record Filtering & Search:
  • Match field contents against strings, numbers, or regular expressions (=, !, <, >, ^, ~, #, ?). Numerical comparison is used for < and > when both operands are numbers, otherwise strcmp.
  • Read comparison values from external files (file:list.txt).
  • Combine expressions with logical OR or AND (-a), with optional case-insensitivity (-X).
• Data Anonymization Engine: Built-in non-reversible anonymization powered by Libgcrypt (MASK, RANDOM, NRANDOM, HASH, NHASH), preserving original field lengths and data types.
• External Pipes & Lookups: Pipe individual field values through external commands (filter) or translate codes using lookup tables (lookup).
• Automatic Structure Guessing: Automatically identifies the matching structure definition when -s is omitted.
• Transparent Preprocessing: Specify input preprocessors via the FFEOPEN environment variable (e.g., lesspipe to process compressed .gz or .bz2 files directly).
• Command Substitution: Embed dynamic values in configuration files using backtick syntax (`command`).

---

Installation

Prerequisites

To build ffe, you need a C99 compiler, GNU Autotools, and Libgcrypt:

• Debian / Ubuntu:

  sudo apt-get install build-essential autoconf automake texinfo libgcrypt20-dev

• Fedora / RHEL / Rocky Linux:

  sudo dnf install gcc make autoconf automake texinfo libgcrypt-devel

• macOS (Homebrew):

  brew install autoconf automake texinfo libgcrypt

Building from Source

git clone https://github.com/TimoSavi/ffe.git
cd ffe
autoreconf -is
./configure
make
sudo make install

To run the test suite and verify the build:

make check

---

Quick Start Tutorial

Configuration is read by default from ~/.fferc (or from a custom file using -c).

1. Converting Fixed-Length Records to XML

Suppose you have a fixed-length file named personnel:

john     Ripper       23
Scott    Tiger        45
Mary     Moore        41

Define the structure and desired XML output in ~/.fferc:

structure personnel_file {
    type fixed
    record person {
        field Firstname  9
        field Lastname  13
        field Age        2
    }
}

output xml_format {
    file_header "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<people>\n"
    file_trailer "</people>\n"
    record_header "  <person>\n"
    record_trailer "  </person>\n"
    data "    <%n>%d</%n>\n"
}

Run ffe:

ffe -s personnel_file -p xml_format personnel

Output:

<?xml version="1.0" encoding="UTF-8"?>
<people>
  <person>
    <Firstname>john</Firstname>
    <Lastname>Ripper</Lastname>
    <Age>23</Age>
  </person>
  <person>
    <Firstname>Scott</Firstname>
    <Lastname>Tiger</Lastname>
    <Age>45</Age>
  </person>
  <person>
    <Firstname>Mary</Firstname>
    <Lastname>Moore</Lastname>
    <Age>41</Age>
  </person>
</people>

---

2. Parsing Delimited Files (CSV / TSV)

For a comma-separated file data.csv:

101,John Doe,Developer,75000
102,Jane Smith,Manager,90000

Add this structure to your config:

structure employee_csv {
    type separated ,
    record employee {
        field Id
        field Name
        field Role
        field Salary
    }
}

Print specific fields:

ffe -s employee_csv -f Name,Role data.csv

---

3. Filtering Records with Expressions

Select records using the -e / --expression option (field<op>value):

Operator	Meaning	Description
=	Equal	Field equals value
!	Not equal	Field does not equal value
<	Less than	Numeric comparison if both sides are numbers, otherwise strcmp
>	Greater than	Numeric comparison if both sides are numbers, otherwise strcmp
^	Starts with	Field starts with value
~	Contains	Field contains substring
#	Does not contain	Field does not contain substring
?	Regexp	POSIX extended regular expression match

Note

Comparison operators (< and >) do not support multiple values for the same field or file: input. Range filtering is achieved by combining expressions with -a (e.g., -a -e "Age>25" -e "Age<45").

Examples:

• Records where Age is greater than 30:

  ffe -s personnel_file -e "Age>30" personnel

• Case-insensitive substring match (-X and ~):

  ffe -s personnel_file -X -e "Lastname~ripper" personnel

• Match values against an external file:

  ffe -s personnel_file -e "Lastname=file:target_names.txt" personnel

• Combine conditions with logical AND (-a) for ranges:

  ffe -s personnel_file -a -e "Age>25" -e "Age<45" personnel

---

4. Data Anonymization

ffe includes a non-reversible anonymizer suitable for GDPR/PCI-DSS compliance and testing data generation.

In ~/.fferc:

anonymize safe_export {
    method Firstname RANDOM
    method Lastname  RANDOM
    method Age       MASK 0
}

Run ffe with -A:

ffe -s personnel_file -A safe_export personnel

---

Command-Line Options

Option	Long Option	Description
-c FILE	--configuration=FILE	Read configuration from FILE (default: ~/.fferc).
-s STRUCTURE	--structure=STRUCTURE	Use structure STRUCTURE (disables auto-guessing).
-p FORMAT	--print=FORMAT	Use output format FORMAT for printing.
-o FILE	--output=FILE	Write output to FILE instead of standard output.
-f LIST	--field-list=LIST	Print only comma-separated fields/constants in LIST.
-e EXPR	--expression=EXPR	Filter records matching expression (field<op>value).
-a	--and	Combine multiple -e expressions using AND (default: OR).
-X	--casecmp	Evaluate expressions case-insensitively.
-v	--invert-match	Invert expression match (print non-matching records).
-l	--loose	Continue processing even if an invalid line is encountered.
-r F=V	--replace=FIELD=VAL	Replace FIELD's content with VAL in the output.
-d	--debug	Write invalid input lines to ffe_error_<pid>.log.
-I	--info	Display structure information from config file and exit.
-A NAME	--anonymize=NAME	Apply anonymization profile NAME.
-?	--help	Display command-line help and exit.
-V	--version	Display version information and exit.

---

Configuration Overview

Structures

A structure defines how input data is partitioned into records:

structure <name> {
    type fixed | separated <delimiter> | binary
    quoted yes | no | <char>
    header first | all | no
    output <output_name>
    record <record_name> { ... }
}

Records

A record defines fields and identification criteria:

record <name> {
    id <position> <string>        # Exact string identifier
    rid <position> <regexp>       # Regular expression identifier
    field <name> [<length>|<type>] [<lookup>] [<output>] [<filter>] [<conversion>]
    field FILLER <length>         # Ignored/padding bytes
}

Outputs

An output format governs how extracted fields are displayed:

output <name> {
    file_header <picture>         # Printed once before file data
    file_trailer <picture>        # Printed once after file data
    record_header <picture>       # Printed before each record
    record_trailer <picture>      # Printed after each record (default: newline)
    data <picture>                # Printed for each field
    lookup <picture>              # Used if field references a lookup table
    separator <string>            # Field delimiter
    indent <string>               # Indentation string
    field-list <name1>,<name2>    # Field subset and ordering
    hex-caps yes | no             # Uppercase hex numbers
}

Output Format Directives

Inside output pictures (like data or record_header), the following % directives can be used:

Directive	Description
%d	Field contents formatted according to its type (binary as decimal, float as [-]ddd.ddd, BCD as decimal, hex as hex).
%t	Field contents with leading and trailing whitespace trimmed.
%D	Field contents right-padded with spaces to the field length.
%C	Field contents right-padded to field length; truncated if longer.
%n	Name of the current field.
%l	Lookup value found using the current field as search key.
%L	Lookup value right-padded with spaces to the field length.
%p	Field start position (byte position for fixed/binary, field index for separated, starting from 1).
%x	Unsigned hexadecimal value of a binary integer.
%h	Hexadecimal byte dump of a binary field (xnn representation).
%e	Suppresses output but triggers field empty check.
%s	Name of the current structure.
%r	Name of the current record.
%o	Input record number in current file (starting from 1).
%O	Input record number starting from the first file.
%i	Byte offset of the current record in current file (starting from 0).
%I	Byte offset of the current record starting from the first file.
%g	Group name (defined by group_name for hierarchical levels).
%m	Element name (defined by element_name for hierarchical levels).
%%	Literal percent sign (%).
\n, \t, \r, \"	Standard escape sequences.

Anonymization

Anonymization rules are defined with the anonymize block:

anonymize <name> {
    method <field> <algorithm> [<start>] [<length>] [<parameter>]
}

Supported algorithms:

• MASK: Replace characters with a masking character (default: '0').
• RANDOM: Replace with pseudo-random characters (0-9, a-z, A-Z, space).
• NRANDOM: Replace with pseudo-random numeric digits (0-9).
• HASH: Replace with cryptographic hash digest (Libgcrypt MD5/SHA) maintaining determinism for matching inputs.
• NHASH: Replace with numeric cryptographic hash.

Lookup Tables and Filters

• Lookup Tables:

  lookup country_code {
      pair "FI" "Finland"
      pair "SE" "Sweden"
      default "Unknown"
  }
  

• Shell Filters:

  filter uppercase "tr '[:lower:]' '[:upper:]'"
  

---

Documentation & Future Conversions

Reading the Manual

The authoritative, comprehensive user guide is maintained in GNU Texinfo format in doc/ffe.texi.

When ffe is built, the Texinfo source is compiled into standard formats:

• GNU Info: info ffe (or view doc/ffe.info)
• HTML Manual: Open doc/ffe.html in any browser
• Man Page: man ffe (source: doc/ffe.1)

How to Convert doc/ffe.texi to Markdown

If you update doc/ffe.texi in the future and want to regenerate or update this README.md, you can use either of the following methods:

Method A: Automated Conversion via texi2any and pandoc

GNU Texinfo does not natively output Markdown. The standard conversion pipeline converts Texinfo to DocBook XML first, then translates DocBook into GitHub-Flavored Markdown:

1. Install tools (if not already present):

   sudo apt-get install texinfo pandoc     # Debian/Ubuntu
   sudo dnf install texinfo pandoc         # Fedora/RHEL

2. Run the conversion pipeline:

   # Step 1: Export Texinfo source to DocBook XML
   texi2any --docbook doc/ffe.texi -o doc/ffe.xml
   
   # Step 2: Convert DocBook XML to GitHub-Flavored Markdown
   pandoc -f docbook -t gfm doc/ffe.xml -o README.md
   
   # Step 3: Clean up intermediate XML
   rm doc/ffe.xml

Method B: Curated Extraction (Recommended for GitHub)

While automated conversion via pandoc produces complete output, it often includes Texinfo menu artifacts, navigation headers, and rigid formatting. For a project README.md, manually or AI-assisted updating from doc/ffe.texi is recommended to ensure:

• Clean GitHub badges and concise tables.
• Modern syntax highlighting blocks (sh`, xml, ````text).
• Direct, task-focused quick start examples.

---

License & Author

ffe was written by Timo Savinen (tjsa@iki.fi).

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. See the COPYING file for full details.