App settings management guide

Settings for the redaction app can be set from outside by changing values in the .env file stored in your local config folder, or in S3 if running on AWS. This guide provides an overview of how to configure the application using environment variables. The application loads configurations using os.environ.get(). It first attempts to load variables from the file specified by APP_CONFIG_PATH (which defaults to config/app_config.env). If AWS_CONFIG_PATH is also set (e.g., to config/aws_config.env), variables are loaded from that file as well. Environment variables set directly in the system will always take precedence over those defined in these .env files.

Value Format Notes

  • Boolean Values: Boolean environment variables accept multiple formats: "True", "1", "true", or "TRUE" for true; "False", "0", "false", or "FALSE" for false.

  • List Values: List environment variables should be provided as comma-separated strings within square brackets, e.g., "['item1', 'item2', 'item3']". The application will automatically parse these into Python lists.

  • Temporary Folders: Setting OUTPUT_FOLDER or INPUT_FOLDER to "TEMP" will create a temporary directory that is automatically cleaned up when the application exits. This is useful for increased security in some deployment scenarios.

App Configuration File (config.env)

This section details variables related to the main application configuration file.

  • CONFIG_FOLDER
    • Description: The folder where configuration files are stored.
    • Default Value: config/
  • APP_CONFIG_PATH
    • Description: Specifies the path to the application configuration .env file. This file contains various settings that control the application’s behavior.
    • Default Value: config/app_config.env

AWS Options

This section covers configurations related to AWS services used by the application.

  • AWS_CONFIG_PATH
    • Description: Specifies the path to the AWS configuration .env file. This file is intended to store AWS credentials and specific settings.
    • Default Value: '' (empty string)
  • RUN_AWS_FUNCTIONS
    • Description: Enables or disables AWS-specific functionalities within the application. Set to "True" to enable.
    • Default Value: "False"
  • AWS_REGION
    • Description: Defines the AWS region where services like S3, Cognito, and Textract are located.
    • Default Value: ''
  • AWS_CLIENT_ID
    • Description: The client ID for AWS Cognito, used for user authentication.
    • Default Value: ''
  • AWS_CLIENT_SECRET
    • Description: The client secret for AWS Cognito, used in conjunction with the client ID for authentication.
    • Default Value: ''
  • AWS_USER_POOL_ID
    • Description: The user pool ID for AWS Cognito, identifying the user directory.
    • Default Value: ''
  • AWS_ACCESS_KEY
    • Description: The AWS access key ID for programmatic access to AWS services.
    • Default Value: ''
  • AWS_SECRET_KEY
    • Description: The AWS secret access key corresponding to the AWS access key ID.
    • Default Value: ''
  • DOCUMENT_REDACTION_BUCKET
    • Description: The name of the S3 bucket used for storing documents related to the redaction process.
    • Default Value: ''
  • PRIORITISE_SSO_OVER_AWS_ENV_ACCESS_KEYS
    • Description: If set to "True", the app will prioritize using AWS SSO credentials over access keys stored in environment variables.
    • Default Value: "True"
  • CUSTOM_HEADER
    • Description: Specifies a custom header name to be included in requests, often used for services like AWS CloudFront.
    • Default Value: ''
  • CUSTOM_HEADER_VALUE
    • Description: The value for the custom header specified by CUSTOM_HEADER.
    • Default Value: ''

Image Options

Settings related to image processing within the application.

  • IMAGES_DPI
    • Description: Dots Per Inch (DPI) setting for image processing, affecting the resolution and quality of processed images.
    • Default Value: '300.0'
  • LOAD_TRUNCATED_IMAGES
    • Description: Controls whether the application attempts to load truncated images. Set to 'True' to enable.
    • Default Value: 'True'
  • MAX_IMAGE_PIXELS
    • Description: Sets the maximum number of pixels for an image that the application will process. Leave blank for no limit. This can help prevent issues with very large images.
    • Default Value: ''

File I/O Options

Configuration for input and output file handling.

  • SESSION_OUTPUT_FOLDER
    • Description: If set to 'True', the application will save output and input files into session-specific subfolders.
    • Default Value: 'False'
  • OUTPUT_FOLDER (environment variable: GRADIO_OUTPUT_FOLDER)
    • Description: Specifies the default output folder for generated files. Can be set to "TEMP" to use a temporary directory.
    • Default Value: 'output/'
  • INPUT_FOLDER (environment variable: GRADIO_INPUT_FOLDER)
    • Description: Specifies the default input folder for files. Can be set to "TEMP" to use a temporary directory.
    • Default Value: 'input/'
  • GRADIO_TEMP_DIR
    • Description: Defines the path for Gradio’s temporary file storage.
    • Default Value: ''
  • MPLCONFIGDIR
    • Description: Specifies the cache directory for the Matplotlib library.
    • Default Value: ''

Logging Options

Settings for configuring application logging.

Note: By default, logs are stored in subfolders based on today’s date and the hostname of the instance running the app (e.g., logs/20240101/hostname/). This prevents log files from one instance overwriting logs from another instance, which is especially important when running on S3 or in multi-instance deployments. If you’re always running the app on a single system or just locally, you can disable this behavior by setting USE_LOG_SUBFOLDERS to "False".

  • SAVE_LOGS_TO_CSV
    • Description: Enables or disables saving logs to CSV files. Set to 'True' to enable.
    • Default Value: 'True'
  • USE_LOG_SUBFOLDERS
    • Description: If enabled ('True'), logs will be stored in subfolders based on date and hostname.
    • Default Value: 'True'
  • FEEDBACK_LOGS_FOLDER, ACCESS_LOGS_FOLDER, USAGE_LOGS_FOLDER
    • Description: Base folders for feedback, access, and usage logs respectively.
    • Default Values: 'feedback/', 'logs/', 'usage/'
  • S3_FEEDBACK_LOGS_FOLDER, S3_ACCESS_LOGS_FOLDER, S3_USAGE_LOGS_FOLDER
    • Description: S3 paths where feedback, access, and usage logs will be stored if RUN_AWS_FUNCTIONS is enabled.
    • Default Values: Dynamically generated based on date and hostname, e.g., 'feedback/YYYYMMDD/hostname/'.
  • LOG_FILE_NAME, USAGE_LOG_FILE_NAME, FEEDBACK_LOG_FILE_NAME
    • Description: Specifies the name for log files. USAGE_LOG_FILE_NAME and FEEDBACK_LOG_FILE_NAME default to the value of LOG_FILE_NAME.
    • Default Value: 'log.csv'
  • DISPLAY_FILE_NAMES_IN_LOGS
    • Description: If set to 'True', file names will be included in log entries.
    • Default Value: 'False'
  • CSV_ACCESS_LOG_HEADERS, CSV_FEEDBACK_LOG_HEADERS, CSV_USAGE_LOG_HEADERS
    • Description: Defines custom headers for the respective CSV logs as a string representation of a list. If blank, component labels are used.
    • Default Value: Varies; see script for CSV_USAGE_LOG_HEADERS default.
  • SAVE_LOGS_TO_DYNAMODB
    • Description: Enables or disables saving logs to AWS DynamoDB. Set to 'True' to enable.
    • Default Value: 'False'
  • ACCESS_LOG_DYNAMODB_TABLE_NAME, FEEDBACK_LOG_DYNAMODB_TABLE_NAME, USAGE_LOG_DYNAMODB_TABLE_NAME
    • Description: Names of the DynamoDB tables for storing access, feedback, and usage logs.
    • Default Values: 'redaction_access_log', 'redaction_feedback', 'redaction_usage'
  • DYNAMODB_ACCESS_LOG_HEADERS, DYNAMODB_FEEDBACK_LOG_HEADERS, DYNAMODB_USAGE_LOG_HEADERS
    • Description: Specifies the headers (attributes) for the respective DynamoDB log tables.
    • Default Value: ''
  • LOGGING
    • Description: Enables or disables general console logging. Set to 'True' to enable.
    • Default Value: 'False'

Gradio & General App Options

Configurations for the Gradio UI, server behavior, and application limits.

  • FAVICON_PATH
    • Description: Path to the favicon icon file for the web interface.
    • Default Value: "favicon.png"
  • RUN_FASTAPI
    • Description: If set to "True", the application will be served via FastAPI, allowing for API endpoint integration.
    • Default Value: "False"
  • RUN_MCP_SERVER
    • Description: If set to "True", the application will run as an MCP (Model Context Protocol) server.
    • Default Value: "False"
  • GRADIO_SERVER_NAME
    • Description: The IP address the Gradio server will bind to. Use "0.0.0.0" to allow external access.
    • Default Value: "127.0.0.1"
  • GRADIO_SERVER_PORT
    • Description: The network port on which the Gradio server will listen.
    • Default Value: 7860
  • ALLOWED_ORIGINS
    • Description: A comma-separated list of allowed origins for Cross-Origin Resource Sharing (CORS).
    • Default Value: ''
  • ALLOWED_HOSTS
    • Description: A comma-separated list of allowed hostnames.
    • Default Value: ''
  • ROOT_PATH
    • Description: The root path for the application, useful if running behind a reverse proxy (e.g., /app).
    • Default Value: ''
  • FASTAPI_ROOT_PATH
    • Description: The root path for the FastAPI application, used when RUN_FASTAPI is true.
    • Default Value: "/"
  • MAX_QUEUE_SIZE
    • Description: The maximum number of requests that can be queued in the Gradio interface.
    • Default Value: 5
  • MAX_FILE_SIZE
    • Description: Maximum file size allowed for uploads (e.g., “250mb”, “1gb”).
    • Default Value: '250mb'
  • DEFAULT_CONCURRENCY_LIMIT
    • Description: The default concurrency limit for Gradio event handlers, controlling how many requests can be processed simultaneously.
    • Default Value: 3
  • MAX_SIMULTANEOUS_FILES
    • Description: The maximum number of files that can be processed at once.
    • Default Value: 10
  • MAX_DOC_PAGES
    • Description: The maximum number of pages a document can have.
    • Default Value: 3000
  • MAX_TABLE_ROWS / MAX_TABLE_COLUMNS
    • Description: Maximum number of rows and columns for tabular data processing.
    • Default Values: 250000 / 100
  • MAX_OPEN_TEXT_CHARACTERS
    • Description: Maximum number of characters for open text input.
    • Default Value: 50000
  • PAGE_BREAK_VALUE
    • Description: Number of pages to process before breaking and restarting from the last finished page.
    • Default Value: 99999
    • Note: This feature is not currently activated in the application.
  • MAX_TIME_VALUE
    • Description: Maximum time value for processing operations.
    • Default Value: 999999
  • TLDEXTRACT_CACHE
    • Description: Path to the cache directory used by the tldextract library.
    • Default Value: 'tmp/tld/'
  • COGNITO_AUTH
    • Description: Enables or disables AWS Cognito authentication. Set to 'True' to enable.
    • Default Value: 'False'
  • SHOW_FEEDBACK_BUTTONS
    • Description: If set to "True", displays feedback buttons in the Gradio interface.
    • Default Value: "False"
  • USER_GUIDE_URL
    • Description: A safe URL pointing to the user guide. The URL is validated against a list of allowed domains.
    • Default Value: "https://seanpedrick-case.github.io/doc_redaction"
  • INTRO_TEXT
    • Description: Custom introduction text for the app. Should be in Markdown format, html is stripped out. Can also be set to a path to a .txt file (e.g., "intro.txt"), which will be read and used as the intro text. The text is automatically sanitized to remove dangerous HTML/scripts while preserving safe markdown syntax.
    • Default Value: "Too long to display here, see tools/config.py"
  • SHOW_EXAMPLES
    • Description: If set to "True", displays example files in the Gradio interface.
    • Default Value: "True"
  • SHOW_AWS_EXAMPLES
    • Description: If set to "True", includes AWS-specific examples.
    • Default Value: "False"
  • SHOW_DIFFICULT_OCR_EXAMPLES
    • Description: If set to "True", includes examples that demonstrate difficult OCR scenarios.
    • Default Value: "False"
  • SHOW_ALL_OUTPUTS_IN_OUTPUT_FOLDER
    • Description: If set to "True", displays all output files in the output folder interface.
    • Default Value: "False"
  • FILE_INPUT_HEIGHT
    • Description: Sets the height (in pixels) of the file input component in the Gradio UI.
    • Default Value: 200

Redaction & PII Options

Configurations related to text extraction, PII detection, and the redaction process.

UI and Model Selection

  • EXTRACTION_AND_PII_OPTIONS_OPEN_BY_DEFAULT
    • Description: If set to "True", the “Extraction and PII Options” accordion in the UI will be open by default.
    • Default Value: "True"
  • SHOW_LOCAL_TEXT_EXTRACTION_OPTIONS / SHOW_AWS_TEXT_EXTRACTION_OPTIONS
    • Description: Controls whether local (Tesseract) or AWS (Textract) text extraction options are shown in the UI.
    • Default Value: "True" for both.
    • Note: If both are set to "False", the application will automatically enable SHOW_LOCAL_TEXT_EXTRACTION_OPTIONS to ensure at least one option is available.
  • SELECTABLE_TEXT_EXTRACT_OPTION, TESSERACT_TEXT_EXTRACT_OPTION, TEXTRACT_TEXT_EXTRACT_OPTION
    • Description: Labels for text extraction model options displayed in the UI. Customize the display names for “Local model - selectable text”, “Local OCR model - PDFs without selectable text”, and “AWS Textract service - all PDF types” respectively.
    • Default Values: "Local model - selectable text", "Local OCR model - PDFs without selectable text", "AWS Textract service - all PDF types"
  • NO_REDACTION_PII_OPTION, LOCAL_PII_OPTION, AWS_PII_OPTION
    • Description: Labels for PII detection model options displayed in the UI. Customize the display names for “Only extract text (no redaction)”, “Local”, and “AWS Comprehend” respectively.
    • Default Values: "Only extract text (no redaction)", "Local", "AWS Comprehend"
  • SHOW_LOCAL_PII_DETECTION_OPTIONS / SHOW_AWS_PII_DETECTION_OPTIONS
    • Description: Controls whether local or AWS (Comprehend) PII detection options are shown in the UI.
    • Default Value: "True" for both.
    • Note: If both are set to "False", the application will automatically enable SHOW_LOCAL_PII_DETECTION_OPTIONS to ensure at least one option is available.
  • DEFAULT_TEXT_EXTRACTION_MODEL
    • Description: Sets the default text extraction model selected in the UI.
    • Default Value: Automatically defaults to AWS Textract if SHOW_AWS_TEXT_EXTRACTION_OPTIONS is enabled, otherwise defaults to the local selectable text option.
  • DEFAULT_PII_DETECTION_MODEL
    • Description: Sets the default PII detection model selected in the UI.
    • Default Value: Automatically defaults to AWS Comprehend if SHOW_AWS_PII_DETECTION_OPTIONS is enabled, otherwise defaults to the local model.
  • LOAD_REDACTION_ANNOTATIONS_FROM_PDF
    • Description: If set to "True", the application will load existing redaction annotations from PDFs during the review step.
    • Default Value: "True"

External Tool Paths

  • TESSERACT_FOLDER
    • Description: Path to the local Tesseract OCR installation folder.
    • Default Value: ''
    • Installation Note: For Windows, install Tesseract 5.5.0 from UB-Mannheim/tesseract. This environment variable should point to the Tesseract folder (e.g., tesseract/).
  • TESSERACT_DATA_FOLDER
    • Description: Path to the Tesseract trained data files (tessdata).
    • Default Value: "/usr/share/tessdata"
  • POPPLER_FOLDER
    • Description: Path to the local Poppler installation’s bin folder.
    • Default Value: ''
    • Installation Note: For Windows, install Poppler from oschwartz10612/poppler-windows. This variable needs to point to the Poppler bin folder (e.g., poppler/poppler-24.02.0/Library/bin/).
  • PADDLE_MODEL_PATH / SPACY_MODEL_PATH
    • Description: Custom directory for PaddleOCR and spaCy model storage, useful for environments like AWS Lambda.
    • Default Value: '' (uses default location).
  • PADDLE_FONT_PATH
    • Description: Custom font path for PaddleOCR. If empty, the application will attempt to use system fonts to avoid downloading simfang.ttf or PingFang-SC-Regular.ttf. Set this if you want to use a specific font file for PaddleOCR text rendering.
    • Default Value: '' (uses system fonts).

Local OCR (Tesseract & PaddleOCR)

  • CHOSEN_LOCAL_OCR_MODEL
    • Description: Choose the engine for local OCR: "tesseract", "paddle", "hybrid-paddle", "hybrid-vlm", "hybrid-paddle-vlm", "hybrid-paddle-inference-server", "vlm", or "inference-server".
    • Default Value: "tesseract"
  • SHOW_LOCAL_OCR_MODEL_OPTIONS
    • Description: If set to "True", allows the user to select the local OCR model from the UI.
    • Default Value: "False"
  • HYBRID_OCR_CONFIDENCE_THRESHOLD
    • Description: In “hybrid-paddle” mode, this is the Tesseract confidence score below which PaddleOCR will be used for re-extraction.
    • Default Value: 80
  • HYBRID_OCR_PADDING
    • Description: In “hybrid-paddle” mode, padding added to the word’s bounding box before re-extraction.
    • Default Value: 1
  • PADDLE_USE_TEXTLINE_ORIENTATION
    • Description: Toggles textline orientation detection for PaddleOCR.
    • Default Value: "False"
  • PADDLE_DET_DB_UNCLIP_RATIO
    • Description: Controls the expansion ratio of the detected text region in PaddleOCR.
    • Default Value: 1.2
  • SAVE_EXAMPLE_HYBRID_IMAGES
    • Description: Saves comparison images when using “hybrid-paddle” OCR mode.
    • Default Value: "False"
  • SAVE_PAGE_OCR_VISUALISATIONS
    • Description: If set to "True", saves visualisations of Tesseract, PaddleOCR, and Textract bounding boxes overlaid on the page images.
    • Default Value: "False"
  • INCLUDE_OCR_VISUALISATION_IN_OUTPUT_FILES
    • Description: If set to "True", OCR visualisation output files (created when SAVE_PAGE_OCR_VISUALISATIONS is enabled) will be included in the final output file list returned by choose_and_run_redactor. This makes the visualisation files available in the Gradio output interface.
    • Default Value: "False"
  • SAVE_WORD_SEGMENTER_OUTPUT_IMAGES
    • Description: If set to "True", saves output images from the word segmenter for debugging purposes.
    • Default Value: "False"
  • PREPROCESS_LOCAL_OCR_IMAGES
    • Description: If set to "True", images will be preprocessed before local OCR. Can slow down processing.
    • Default Value: "True"
    • Note: Testing has shown that preprocessing doesn’t necessarily improve OCR results and can significantly slow down extraction. Consider setting this to "False" if processing speed is a priority.
  • SAVE_PREPROCESS_IMAGES
    • Description: If set to "True", saves the preprocessed images for debugging purposes.
    • Default Value: "False"
  • SHOW_PADDLE_MODEL_OPTIONS
    • Description: If set to "True", allows the user to select PaddleOCR-related options (paddle, hybrid-paddle) from the UI.
    • Default Value: "False"
  • MODEL_CACHE_PATH
    • Description: Path to the directory where models are cached.
    • Default Value: "./model_cache"
  • TESSERACT_SEGMENTATION_LEVEL
    • Description: Tesseract PSM (Page Segmentation Mode) level to use for OCR. Valid values are 0-13.
    • Default Value: 11
  • TESSERACT_WORD_LEVEL_OCR
    • Description: If set to "True", uses Tesseract word-level OCR instead of line-level.
    • Default Value: "True"
  • CONVERT_LINE_TO_WORD_LEVEL
    • Description: If set to "True", converts PaddleOCR line-level OCR results to word-level for better precision.
    • Default Value: "False"
  • LOAD_PADDLE_AT_STARTUP
    • Description: If set to "True", loads the PaddleOCR model at application startup.
    • Default Value: "False"
  • SHOW_INFERENCE_SERVER_OPTIONS
    • Description: If set to "True", allows the user to select inference-server-related options from the UI.
    • Default Value: "False"
  • SHOW_HYBRID_MODELS
    • Description: If set to "True", enables hybrid model options (e.g., hybrid-paddle-vlm, hybrid-paddle-inference-server) in the UI.
    • Default Value: "False"
  • INFERENCE_SERVER_API_URL
    • Description: Base URL of the inference-server API for remote OCR processing.
    • Default Value: "http://localhost:8080"
  • INFERENCE_SERVER_MODEL_NAME
    • Description: Optional model name to use for inference-server API. If empty, uses the default model on the server.
    • Default Value: ''
  • INFERENCE_SERVER_TIMEOUT
    • Description: Timeout in seconds for inference-server API requests.
    • Default Value: 300

Vision Language Model (VLM) Options

  • SHOW_VLM_MODEL_OPTIONS
    • Description: If set to "True", VLM (Vision Language Model) options will be shown in the UI.
    • Default Value: "False"
  • SELECTED_MODEL
    • Description: Selected vision model for OCR. Choose from: "Nanonets-OCR2-3B", "Dots.OCR", "Qwen3-VL-2B-Instruct", "Qwen3-VL-4B-Instruct", "Qwen3-VL-8B-Instruct", "PaddleOCR-VL".
    • Default Value: "Qwen3-VL-4B-Instruct"
  • QUANTISE_VLM_MODELS
    • Description: If set to "True", the VLM models will be quantized using 4-bit quantisation (bitsandbytes).
    • Default Value: "False"
  • USE_FLASH_ATTENTION
    • Description: If set to "True", uses flash attention for the VLM, which can improve performance (not possible on Windows).
    • Default Value: "False"
  • VLM_MAX_IMAGE_SIZE
    • Description: Maximum total pixels (width * height) for images passed to VLM. Images with more pixels will be resized while maintaining aspect ratio. A multiple of 32*32 for Qwen3-VL.
    • Default Value: 819200 (approximately 800x800)
  • VLM_MIN_IMAGE_SIZE
    • Description: Minimum total pixels (width * height) for images passed to VLM. Images with less pixels will be resized while maintaining aspect ratio. A multiple of 32*32 for Qwen3-VL.
    • Default Value: 614400 (approximately 600x600)
  • VLM_MAX_DPI
    • Description: Maximum DPI for images passed to VLM. Images with higher DPI will be resized accordingly.
    • Default Value: 300.0
  • MAX_SPACES_GPU_RUN_TIME
    • Description: Maximum number of seconds to run the GPU on Spaces (Hugging Face Spaces).
    • Default Value: 60
  • MAX_NEW_TOKENS
    • Description: Maximum number of tokens to generate for VLM responses.
    • Default Value: 4192
  • DEFAULT_MAX_NEW_TOKENS
    • Description: Default maximum number of tokens to generate for VLM responses.
    • Default Value: 4192
  • MAX_INPUT_TOKEN_LENGTH
    • Description: Maximum number of tokens that can be input to the VLM.
    • Default Value: 8192
  • OVERWRITE_EXISTING_OCR_RESULTS
    • Description: If set to "True", always creates new OCR results instead of loading from existing JSON files.
    • Default Value: "False"
  • SAVE_VLM_INPUT_IMAGES
    • Description: If set to "True", saves input images sent to VLM OCR for debugging purposes.
    • Default Value: "False"
  • HYBRID_OCR_MAX_NEW_TOKENS
    • Description: Maximum number of tokens to generate for hybrid OCR operations.
    • Default Value: 30
  • REPORT_VLM_OUTPUTS_TO_GUI
    • Description: If set to "True", reports VLM outputs to the GUI with info boxes as they are processed.
    • Default Value: "False"
  • VLM_SEED
    • Description: Random seed for VLM generation. If empty, no seed is set (non-deterministic). If set to an integer, generation will be deterministic.
    • Default Value: "42"
  • VLM_DEFAULT_TEMPERATURE
    • Description: Default temperature for VLM generation. Used when model-specific defaults are not set.
    • Default Value: 0.1
  • VLM_DEFAULT_TOP_P
    • Description: Default top_p (nucleus sampling) for VLM generation. Used when model-specific defaults are not set.
    • Default Value: 0.8
  • VLM_DEFAULT_MIN_P
    • Description: Default min_p (minimum probability threshold) for VLM generation. Used when model-specific defaults are not set.
    • Default Value: 0.0
  • VLM_DEFAULT_TOP_K
    • Description: Default top_k for VLM generation. Used when model-specific defaults are not set.
    • Default Value: 20
  • VLM_DEFAULT_REPETITION_PENALTY
    • Description: Default repetition penalty for VLM generation. Used when model-specific defaults are not set.
    • Default Value: 1.3
  • VLM_DEFAULT_DO_SAMPLE
    • Description: Default do_sample setting for VLM generation. "True" means use sampling (do_sample=True), "False" means use greedy decoding (do_sample=False). Used when model-specific defaults are not set.
    • Default Value: "True"
  • VLM_DEFAULT_PRESENCE_PENALTY
    • Description: Default presence penalty for VLM generation. If empty, defaults to None. Used when model-specific defaults are not set.
    • Default Value: ''

Entity and Search Options

  • CHOSEN_COMPREHEND_ENTITIES / FULL_COMPREHEND_ENTITY_LIST
    • Description: The selected and available PII entity types for AWS Comprehend.
    • Default Value: Predefined lists of entities (see script).
  • CHOSEN_REDACT_ENTITIES / FULL_ENTITY_LIST
    • Description: The selected and available PII entity types for the local model.
    • Default Value: Predefined lists of entities (see script).
  • CUSTOM_ENTITIES
    • Description: A list of entities that are considered “custom” and may have special handling.
    • Default Value: ['TITLES', 'UKPOSTCODE', 'STREETNAME', 'CUSTOM']
  • DEFAULT_SEARCH_QUERY
    • Description: The default text for the custom search/redact input box.
    • Default Value: ''
  • DEFAULT_FUZZY_SPELLING_MISTAKES_NUM
    • Description: Default number of allowed spelling mistakes for fuzzy searches.
    • Default Value: 1
  • DEFAULT_PAGE_MIN / DEFAULT_PAGE_MAX
    • Description: Default start and end pages for processing. 0 for max means process all pages.
    • Default Value: 0 for both.

Textract Feature Selection

  • DEFAULT_HANDWRITE_SIGNATURE_CHECKBOX
    • Description: The default options selected for Textract’s handwriting and signature detection.
    • Default Value: ['Extract handwriting']
  • HANDWRITE_SIGNATURE_TEXTBOX_FULL_OPTIONS
    • Description: Full list of available options for Textract’s handwriting and signature detection. Can include 'Extract handwriting', 'Extract signatures', and optionally 'Extract forms', 'Extract layout', 'Extract tables' if the corresponding include options are enabled.
    • Default Value: ['Extract handwriting', 'Extract signatures']
  • INCLUDE_FORM_EXTRACTION_TEXTRACT_OPTION
  • INCLUDE_LAYOUT_EXTRACTION_TEXTRACT_OPTION
  • INCLUDE_TABLE_EXTRACTION_TEXTRACT_OPTION
    • Description: Booleans ("True"/"False") to include Forms, Layout, and Tables as selectable options for Textract analysis.
    • Default Value: "False" for all.

Tabular Data Options

  • DO_INITIAL_TABULAR_DATA_CLEAN
    • Description: If "True", performs an initial cleaning step on tabular data.
    • Default Value: "True"
  • DEFAULT_TEXT_COLUMNS / DEFAULT_EXCEL_SHEETS
    • Description: Default values for specifying which columns or sheets to process in tabular files.
    • Default Value: [] (empty list)
  • DEFAULT_TABULAR_ANONYMISATION_STRATEGY
    • Description: The default method for anonymizing tabular data (e.g., “redact completely”).
    • Default Value: "redact completely"

Language Options

Settings for multi-language support.

  • SHOW_LANGUAGE_SELECTION
    • Description: If set to "True", a language selection dropdown will be visible in the UI.
    • Default Value: "False"
  • DEFAULT_LANGUAGE_FULL_NAME / DEFAULT_LANGUAGE
    • Description: The default language’s full name (e.g., “english”) and its short code (e.g., “en”).
    • Default Values: "english", "en"
    • Language Support Notes:
      • Tesseract: Ensure the Tesseract language data (e.g., fra.traineddata) is installed. Find language packs at tesseract-ocr/tessdata.
      • PaddleOCR: Ensure the PaddleOCR language data is installed. See supported languages at PaddleOCR documentation.
      • AWS Comprehend: Only English (en) and Spanish (es) are supported. See AWS Comprehend PII documentation.
      • AWS Textract: Automatically detects language and supports English, Spanish, Italian, Portuguese, French, and German. Note that handwriting, invoices, receipts, identity documents, and queries processing are in English only. See AWS Textract FAQs.
  • textract_language_choices / aws_comprehend_language_choices
    • Description: Lists of supported language codes for Textract and Comprehend.
    • Default Value: ['en', 'es', 'fr', 'de', 'it', 'pt'] and ['en', 'es']
  • MAPPED_LANGUAGE_CHOICES / LANGUAGE_CHOICES
    • Description: Paired lists of full language names and their corresponding short codes for the UI dropdown.
    • Default Value: Predefined lists (see script).

Duplicate Detection Settings

  • DEFAULT_DUPLICATE_DETECTION_THRESHOLD
    • Description: The similarity score (0.0 to 1.0) above which documents/pages are considered duplicates.
    • Default Value: 0.95
  • DEFAULT_MIN_CONSECUTIVE_PAGES
    • Description: Minimum number of consecutive pages that must be duplicates to be flagged.
    • Default Value: 1
  • USE_GREEDY_DUPLICATE_DETECTION
    • Description: If "True", uses a greedy algorithm that may find more duplicates but can be less precise.
    • Default Value: "True"
  • DEFAULT_COMBINE_PAGES
    • Description: If "True", text from the same page number across different files is combined before checking for duplicates. If "False", line-level duplicate detection will be enabled instead.
    • Default Value: "True"
  • DEFAULT_MIN_WORD_COUNT
    • Description: Pages with fewer words than this value will be ignored by the duplicate detector.
    • Default Value: 10
  • REMOVE_DUPLICATE_ROWS
    • Description: If "True", enables duplicate row detection in tabular data.
    • Default Value: "False"

File Output Options

  • USE_GUI_BOX_COLOURS_FOR_OUTPUTS
    • Description: If "True", the final redacted PDF will use the same redaction box colors as shown in the review UI.
    • Default Value: "False"
  • CUSTOM_BOX_COLOUR
    • Description: Specifies the color for redaction boxes as an RGB tuple string, e.g., "(0, 0, 0)" for black. Alternatively, you can use the named color "grey" (which maps to RGB (128, 128, 128)).
    • Default Value: "(0, 0, 0)"
  • APPLY_REDACTIONS_IMAGES, APPLY_REDACTIONS_GRAPHICS, APPLY_REDACTIONS_TEXT
    • Description: Advanced control over how redactions are applied to underlying images, vector graphics, and text in the PDF, based on PyMuPDF options. 0 is the default for a standard redaction workflow.
    • Default Value: 0 for all.
    • Detailed Options:
      • APPLY_REDACTIONS_IMAGES: 0 = ignore (default), 1 = completely remove images overlapping redaction annotations, 2 = blank out overlapping pixels, 3 = only remove images that are actually visible. Note: Text in images is effectively removed by the overlapping rectangle shape.
      • APPLY_REDACTIONS_GRAPHICS: 0 = ignore (default), 1 = remove graphics fully contained in redaction annotation, 2 = remove any overlapping vector graphics.
      • APPLY_REDACTIONS_TEXT: 0 = remove all characters whose boundary box overlaps any redaction rectangle (default, complies with legal/data protection intentions), 1 = keep text while redacting graphics/images (does NOT comply with data protection intentions - use at your own risk).
  • RETURN_PDF_FOR_REVIEW
    • Description: If set to "True", a PDF with redaction boxes drawn on it (but text not removed) is generated for the “Review” tab.
    • Default Value: "True"
  • RETURN_REDACTED_PDF
    • Description: If set to 'True', the application will return a fully redacted PDF at the end of the main task.
    • Default Value: "True"
  • COMPRESS_REDACTED_PDF
    • Description: If set to 'True', the redacted PDF output will be compressed.
    • Default Value: "False"
    • Warning: On low memory systems, the compression options in PyMuPDF can cause the app to crash if the PDF is longer than 500 pages or so. Setting this to "False" will save the PDF only with a basic cleaning option enabled, which is more memory-efficient.

Direct Mode & Lambda Configuration

Settings for running the application from the command line (Direct Mode) or as an AWS Lambda function.

Note: Many DIRECT_MODE_* variables inherit their default values from their corresponding non-direct-mode variables if not explicitly set. For example, DIRECT_MODE_LANGUAGE defaults to DEFAULT_LANGUAGE, DIRECT_MODE_IMAGES_DPI defaults to IMAGES_DPI, etc.

Direct Mode

  • RUN_DIRECT_MODE
    • Description: Set to 'True' to enable direct command-line mode.
    • Default Value: 'False'
  • DIRECT_MODE_DEFAULT_USER
    • Description: Default username for CLI requests.
    • Default Value: ''
  • DIRECT_MODE_TASK
    • Description: The task to perform: 'redact' or 'deduplicate'.
    • Default Value: 'redact'
  • DIRECT_MODE_INPUT_FILE / DIRECT_MODE_OUTPUT_DIR
    • Description: Path to the input file and output directory for the task.
    • Default Values: '', output/
  • DIRECT_MODE_DUPLICATE_TYPE
    • Description: Type of duplicate detection for direct mode: 'pages' or 'tabular'.
    • Default Value: 'pages'
  • DIRECT_MODE_LANGUAGE
    • Description: Language for document processing in direct mode.
    • Default Value: Inherits from DEFAULT_LANGUAGE
  • DIRECT_MODE_PII_DETECTOR
    • Description: PII detection method for direct mode.
    • Default Value: Inherits from LOCAL_PII_OPTION
  • DIRECT_MODE_OCR_METHOD
    • Description: OCR method for PDF/image processing in direct mode.
    • Default Value: "Local OCR"
  • DIRECT_MODE_PAGE_MIN / DIRECT_MODE_PAGE_MAX
    • Description: First and last page to process in direct mode. 0 for max means process all pages.
    • Default Values: Inherit from DEFAULT_PAGE_MIN / DEFAULT_PAGE_MAX
  • DIRECT_MODE_IMAGES_DPI
    • Description: DPI for image processing in direct mode.
    • Default Value: Inherits from IMAGES_DPI
  • DIRECT_MODE_CHOSEN_LOCAL_OCR_MODEL
    • Description: Local OCR model choice for direct mode.
    • Default Value: Inherits from CHOSEN_LOCAL_OCR_MODEL
  • DIRECT_MODE_PREPROCESS_LOCAL_OCR_IMAGES
    • Description: If set to "True", preprocesses images before OCR in direct mode.
    • Default Value: Inherits from PREPROCESS_LOCAL_OCR_IMAGES
  • DIRECT_MODE_COMPRESS_REDACTED_PDF
    • Description: If set to "True", compresses the redacted PDF output in direct mode.
    • Default Value: Inherits from COMPRESS_REDACTED_PDF
  • DIRECT_MODE_RETURN_PDF_END_OF_REDACTION
    • Description: If set to "True", returns a PDF at the end of redaction in direct mode.
    • Default Value: Inherits from RETURN_REDACTED_PDF
  • DIRECT_MODE_EXTRACT_FORMS
    • Description: If set to "True", extracts forms during Textract analysis in direct mode.
    • Default Value: "False"
  • DIRECT_MODE_EXTRACT_TABLES
    • Description: If set to "True", extracts tables during Textract analysis in direct mode.
    • Default Value: "False"
  • DIRECT_MODE_EXTRACT_LAYOUT
    • Description: If set to "True", extracts layout during Textract analysis in direct mode.
    • Default Value: "False"
  • DIRECT_MODE_EXTRACT_SIGNATURES
    • Description: If set to "True", extracts signatures during Textract analysis in direct mode.
    • Default Value: "False"
  • DIRECT_MODE_MATCH_FUZZY_WHOLE_PHRASE_BOOL
    • Description: If set to "True", matches fuzzy whole phrases in direct mode.
    • Default Value: "True"
  • DIRECT_MODE_ANON_STRATEGY
    • Description: Anonymisation strategy for tabular data in direct mode.
    • Default Value: Inherits from DEFAULT_TABULAR_ANONYMISATION_STRATEGY
  • DIRECT_MODE_FUZZY_MISTAKES
    • Description: Number of fuzzy spelling mistakes allowed in direct mode.
    • Default Value: Inherits from DEFAULT_FUZZY_SPELLING_MISTAKES_NUM
  • DIRECT_MODE_SIMILARITY_THRESHOLD
    • Description: Similarity threshold for duplicate detection in direct mode.
    • Default Value: Inherits from DEFAULT_DUPLICATE_DETECTION_THRESHOLD
  • DIRECT_MODE_MIN_WORD_COUNT
    • Description: Minimum word count for duplicate detection in direct mode.
    • Default Value: Inherits from DEFAULT_MIN_WORD_COUNT
  • DIRECT_MODE_MIN_CONSECUTIVE_PAGES
    • Description: Minimum consecutive pages for duplicate detection in direct mode.
    • Default Value: Inherits from DEFAULT_MIN_CONSECUTIVE_PAGES
  • DIRECT_MODE_GREEDY_MATCH
    • Description: If set to "True", uses greedy matching for duplicate detection in direct mode.
    • Default Value: Inherits from USE_GREEDY_DUPLICATE_DETECTION
  • DIRECT_MODE_COMBINE_PAGES
    • Description: If set to "True", combines pages for duplicate detection in direct mode.
    • Default Value: Inherits from DEFAULT_COMBINE_PAGES
  • DIRECT_MODE_REMOVE_DUPLICATE_ROWS
    • Description: If set to "True", removes duplicate rows in tabular data in direct mode.
    • Default Value: Inherits from REMOVE_DUPLICATE_ROWS
  • DIRECT_MODE_TEXTRACT_ACTION
    • Description: Textract action for batch operations in direct mode.
    • Default Value: ''
  • DIRECT_MODE_JOB_ID
    • Description: Job ID for Textract operations in direct mode.
    • Default Value: ''

Lambda Configuration

  • LAMBDA_POLL_INTERVAL
    • Description: Polling interval in seconds for checking Textract job status.
    • Default Value: 30
  • LAMBDA_MAX_POLL_ATTEMPTS
    • Description: Maximum number of polling attempts before timeout.
    • Default Value: 120
  • LAMBDA_PREPARE_IMAGES
    • Description: If "True", prepares images for OCR processing within the Lambda environment.
    • Default Value: "True"
  • LAMBDA_EXTRACT_SIGNATURES
    • Description: Enables signature extraction during Textract analysis in Lambda.
    • Default Value: "False"
  • LAMBDA_DEFAULT_USERNAME
    • Description: Default username for operations initiated by Lambda.
    • Default Value: "lambda_user"

Allow, Deny, & Whole Page Redaction Lists

  • GET_DEFAULT_ALLOW_LIST, GET_DEFAULT_DENY_LIST, GET_DEFAULT_WHOLE_PAGE_REDACTION_LIST
    • Description: Booleans ("True"/"False") to enable the use of allow, deny, or whole-page redaction lists.
    • Default Value: "False"
    • Note: GET_DEFAULT_WHOLE_PAGE_REDACTION_LIST is stored as a string value, not converted to boolean (unlike the other two variables).
  • ALLOW_LIST_PATH, DENY_LIST_PATH, WHOLE_PAGE_REDACTION_LIST_PATH
    • Description: Local paths to the respective CSV list files.
    • Default Value: ''
  • S3_ALLOW_LIST_PATH, S3_DENY_LIST_PATH, S3_WHOLE_PAGE_REDACTION_LIST_PATH
    • Description: Paths to the respective list files within the DOCUMENT_REDACTION_BUCKET.
    • Default Value: ''

Cost Code Options

  • SHOW_COSTS
    • Description: If set to 'True', cost-related information will be displayed in the UI.
    • Default Value: 'False'
  • GET_COST_CODES
    • Description: Enables fetching and using cost codes. Set to 'True' to enable.
    • Default Value: 'False'
  • DEFAULT_COST_CODE
    • Description: Specifies a default cost code.
    • Default Value: ''
  • COST_CODES_PATH / S3_COST_CODES_PATH
    • Description: Local or S3 path to a CSV file containing available cost codes.
    • Default Value: ''
    • File Format: The CSV file should contain a single table with two columns and a header row. The first column should contain cost codes, and the second column should contain a name or description for each cost code.
  • ENFORCE_COST_CODES
    • Description: If set to 'True', makes the selection of a cost code mandatory.
    • Default Value: 'False'

Whole Document API Options (Textract Async)

  • SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS
    • Description: Controls whether UI options for asynchronous whole document Textract calls are displayed.
    • Default Value: 'False'
    • Note: This feature is not currently fully implemented in the application.
  • TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET
    • Description: The S3 bucket used for asynchronous Textract analysis.
    • Default Value: ''
  • TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER / ..._OUTPUT_SUBFOLDER
    • Description: Input and output subfolders within the analysis bucket.
    • Default Values: 'input', 'output'
  • LOAD_PREVIOUS_TEXTRACT_JOBS_S3
    • Description: If set to 'True', the application will load data from previous Textract jobs stored in S3.
    • Default Value: 'False'
  • TEXTRACT_JOBS_S3_LOC / TEXTRACT_JOBS_S3_INPUT_LOC
    • Description: S3 subfolders where Textract job output and input are stored.
    • Default Value: 'output', 'input'
  • TEXTRACT_JOBS_LOCAL_LOC
    • Description: The local subfolder for storing Textract job data.
    • Default Value: 'output'
  • DAYS_TO_DISPLAY_WHOLE_DOCUMENT_JOBS
    • Description: Specifies the number of past days for which to display whole document Textract jobs.
    • Default Value: 7
Back to top