App settings management guide
Settings for the redaction app can be set from outside by changing values in the .env file stored in your local config folder, or in S3 if running on AWS. This guide provides an overview of how to configure the application using environment variables. The application loads configurations using os.environ.get(). It first attempts to load variables from the file specified by APP_CONFIG_PATH (which defaults to config/app_config.env). If AWS_CONFIG_PATH is also set (e.g., to config/aws_config.env), variables are loaded from that file as well. Environment variables set directly in the system will always take precedence over those defined in these .env files.
App Configuration File (config.env)
This section details variables related to the main application configuration file.
CONFIG_FOLDER- Description: The folder where configuration files are stored.
- Default Value:
config/
APP_CONFIG_PATH- Description: Specifies the path to the application configuration
.envfile. This file contains various settings that control the application’s behavior. - Default Value:
config/app_config.env
- Description: Specifies the path to the application configuration
AWS Options
This section covers configurations related to AWS services used by the application.
AWS_CONFIG_PATH- Description: Specifies the path to the AWS configuration
.envfile. This file is intended to store AWS credentials and specific settings. - Default Value:
''(empty string)
- Description: Specifies the path to the AWS configuration
RUN_AWS_FUNCTIONS- Description: Enables or disables AWS-specific functionalities within the application. Set to
"True"to enable. - Default Value:
"False"
- Description: Enables or disables AWS-specific functionalities within the application. Set to
AWS_REGION- Description: Defines the AWS region where services like S3, Cognito, and Textract are located.
- Default Value:
''
AWS_CLIENT_ID- Description: The client ID for AWS Cognito, used for user authentication.
- Default Value:
''
AWS_CLIENT_SECRET- Description: The client secret for AWS Cognito, used in conjunction with the client ID for authentication.
- Default Value:
''
AWS_USER_POOL_ID- Description: The user pool ID for AWS Cognito, identifying the user directory.
- Default Value:
''
AWS_ACCESS_KEY- Description: The AWS access key ID for programmatic access to AWS services.
- Default Value:
''
AWS_SECRET_KEY- Description: The AWS secret access key corresponding to the AWS access key ID.
- Default Value:
''
DOCUMENT_REDACTION_BUCKET- Description: The name of the S3 bucket used for storing documents related to the redaction process.
- Default Value:
''
PRIORITISE_SSO_OVER_AWS_ENV_ACCESS_KEYS- Description: If set to
"True", the app will prioritize using AWS SSO credentials over access keys stored in environment variables. - Default Value:
"True"
- Description: If set to
CUSTOM_HEADER- Description: Specifies a custom header name to be included in requests, often used for services like AWS CloudFront.
- Default Value:
''
CUSTOM_HEADER_VALUE- Description: The value for the custom header specified by
CUSTOM_HEADER. - Default Value:
''
- Description: The value for the custom header specified by
Image Options
Settings related to image processing within the application.
IMAGES_DPI- Description: Dots Per Inch (DPI) setting for image processing, affecting the resolution and quality of processed images.
- Default Value:
'300.0'
LOAD_TRUNCATED_IMAGES- Description: Controls whether the application attempts to load truncated images. Set to
'True'to enable. - Default Value:
'True'
- Description: Controls whether the application attempts to load truncated images. Set to
MAX_IMAGE_PIXELS- Description: Sets the maximum number of pixels for an image that the application will process. Leave blank for no limit. This can help prevent issues with very large images.
- Default Value:
''
File I/O Options
Configuration for input and output file handling.
SESSION_OUTPUT_FOLDER- Description: If set to
'True', the application will save output and input files into session-specific subfolders. - Default Value:
'False'
- Description: If set to
OUTPUT_FOLDER- Description: Specifies the default output folder for generated files. Can be set to
"TEMP"to use a temporary directory. - Default Value:
'output/'
- Description: Specifies the default output folder for generated files. Can be set to
INPUT_FOLDER- Description: Specifies the default input folder for files. Can be set to
"TEMP"to use a temporary directory. - Default Value:
'input/'
- Description: Specifies the default input folder for files. Can be set to
GRADIO_TEMP_DIR- Description: Defines the path for Gradio’s temporary file storage.
- Default Value:
''
MPLCONFIGDIR- Description: Specifies the cache directory for the Matplotlib library.
- Default Value:
''
Logging Options
Settings for configuring application logging.
SAVE_LOGS_TO_CSV- Description: Enables or disables saving logs to CSV files. Set to
'True'to enable. - Default Value:
'True'
- Description: Enables or disables saving logs to CSV files. Set to
USE_LOG_SUBFOLDERS- Description: If enabled (
'True'), logs will be stored in subfolders based on date and hostname. - Default Value:
'True'
- Description: If enabled (
FEEDBACK_LOGS_FOLDER,ACCESS_LOGS_FOLDER,USAGE_LOGS_FOLDER- Description: Base folders for feedback, access, and usage logs respectively.
- Default Values:
'feedback/','logs/','usage/'
S3_FEEDBACK_LOGS_FOLDER,S3_ACCESS_LOGS_FOLDER,S3_USAGE_LOGS_FOLDER- Description: S3 paths where feedback, access, and usage logs will be stored if
RUN_AWS_FUNCTIONSis enabled. - Default Values: Dynamically generated based on date and hostname, e.g.,
'feedback/YYYYMMDD/hostname/'.
- Description: S3 paths where feedback, access, and usage logs will be stored if
LOG_FILE_NAME,USAGE_LOG_FILE_NAME,FEEDBACK_LOG_FILE_NAME- Description: Specifies the name for log files.
USAGE_LOG_FILE_NAMEandFEEDBACK_LOG_FILE_NAMEdefault to the value ofLOG_FILE_NAME. - Default Value:
'log.csv'
- Description: Specifies the name for log files.
DISPLAY_FILE_NAMES_IN_LOGS- Description: If set to
'True', file names will be included in log entries. - Default Value:
'False'
- Description: If set to
CSV_ACCESS_LOG_HEADERS,CSV_FEEDBACK_LOG_HEADERS,CSV_USAGE_LOG_HEADERS- Description: Defines custom headers for the respective CSV logs as a string representation of a list. If blank, component labels are used.
- Default Value: Varies; see script for
CSV_USAGE_LOG_HEADERSdefault.
SAVE_LOGS_TO_DYNAMODB- Description: Enables or disables saving logs to AWS DynamoDB. Set to
'True'to enable. - Default Value:
'False'
- Description: Enables or disables saving logs to AWS DynamoDB. Set to
ACCESS_LOG_DYNAMODB_TABLE_NAME,FEEDBACK_LOG_DYNAMODB_TABLE_NAME,USAGE_LOG_DYNAMODB_TABLE_NAME- Description: Names of the DynamoDB tables for storing access, feedback, and usage logs.
- Default Values:
'redaction_access_log','redaction_feedback','redaction_usage'
DYNAMODB_ACCESS_LOG_HEADERS,DYNAMODB_FEEDBACK_LOG_HEADERS,DYNAMODB_USAGE_LOG_HEADERS- Description: Specifies the headers (attributes) for the respective DynamoDB log tables.
- Default Value:
''
LOGGING- Description: Enables or disables general console logging. Set to
'True'to enable. - Default Value:
'False'
- Description: Enables or disables general console logging. Set to
Gradio & General App Options
Configurations for the Gradio UI, server behavior, and application limits.
FAVICON_PATH- Description: Path to the favicon icon file for the web interface.
- Default Value:
"favicon.png"
RUN_FASTAPI- Description: If set to
"True", the application will be served via FastAPI, allowing for API endpoint integration. - Default Value:
"False"
- Description: If set to
GRADIO_SERVER_NAME- Description: The IP address the Gradio server will bind to. Use
"0.0.0.0"to allow external access. - Default Value:
"0.0.0.0"
- Description: The IP address the Gradio server will bind to. Use
GRADIO_SERVER_PORT- Description: The network port on which the Gradio server will listen.
- Default Value:
7860
ALLOWED_ORIGINS- Description: A comma-separated list of allowed origins for Cross-Origin Resource Sharing (CORS).
- Default Value:
''
ALLOWED_HOSTS- Description: A comma-separated list of allowed hostnames.
- Default Value:
''
ROOT_PATH- Description: The root path for the application, useful if running behind a reverse proxy (e.g.,
/app). - Default Value:
''
- Description: The root path for the application, useful if running behind a reverse proxy (e.g.,
FASTAPI_ROOT_PATH- Description: The root path for the FastAPI application, used when
RUN_FASTAPIis true. - Default Value:
"/"
- Description: The root path for the FastAPI application, used when
MAX_QUEUE_SIZE- Description: The maximum number of requests that can be queued in the Gradio interface.
- Default Value:
5
MAX_FILE_SIZE- Description: Maximum file size allowed for uploads (e.g., “250mb”, “1gb”).
- Default Value:
'250mb'
DEFAULT_CONCURRENCY_LIMIT- Description: The default concurrency limit for Gradio event handlers, controlling how many requests can be processed simultaneously.
- Default Value:
3
MAX_SIMULTANEOUS_FILES- Description: The maximum number of files that can be processed at once.
- Default Value:
10
MAX_DOC_PAGES- Description: The maximum number of pages a document can have.
- Default Value:
3000
MAX_TABLE_ROWS/MAX_TABLE_COLUMNS- Description: Maximum number of rows and columns for tabular data processing.
- Default Values:
250000/100
MAX_OPEN_TEXT_CHARACTERS- Description: Maximum number of characters for open text input.
- Default Value:
50000
TLDEXTRACT_CACHE- Description: Path to the cache directory used by the
tldextractlibrary. - Default Value:
'tmp/tld/'
- Description: Path to the cache directory used by the
COGNITO_AUTH- Description: Enables or disables AWS Cognito authentication. Set to
'True'to enable. - Default Value:
'False'
- Description: Enables or disables AWS Cognito authentication. Set to
USER_GUIDE_URL- Description: A safe URL pointing to the user guide. The URL is validated against a list of allowed domains.
- Default Value:
"https://seanpedrick-case.github.io/doc_redaction"
SHOW_EXAMPLES- Description: If set to
"True", displays example files in the Gradio interface. - Default Value:
"True"
- Description: If set to
SHOW_AWS_EXAMPLES- Description: If set to
"True", includes AWS-specific examples. - Default Value:
"False"
- Description: If set to
FILE_INPUT_HEIGHT- Description: Sets the height (in pixels) of the file input component in the Gradio UI.
- Default Value:
200
Redaction & PII Options
Configurations related to text extraction, PII detection, and the redaction process.
UI and Model Selection
EXTRACTION_AND_PII_OPTIONS_OPEN_BY_DEFAULT- Description: If set to
"True", the “Extraction and PII Options” accordion in the UI will be open by default. - Default Value:
"True"
- Description: If set to
SHOW_LOCAL_TEXT_EXTRACTION_OPTIONS/SHOW_AWS_TEXT_EXTRACTION_OPTIONS- Description: Controls whether local (Tesseract) or AWS (Textract) text extraction options are shown in the UI.
- Default Value:
"True"for both.
SHOW_LOCAL_PII_DETECTION_OPTIONS/SHOW_AWS_PII_DETECTION_OPTIONS- Description: Controls whether local or AWS (Comprehend) PII detection options are shown in the UI.
- Default Value:
"True"for both.
DEFAULT_TEXT_EXTRACTION_MODEL- Description: Sets the default text extraction model selected in the UI.
- Default Value: Defaults to AWS Textract if available, otherwise local selectable text.
DEFAULT_PII_DETECTION_MODEL- Description: Sets the default PII detection model selected in the UI.
- Default Value: Defaults to AWS Comprehend if available, otherwise the local model.
LOAD_REDACTION_ANNOTATIONS_FROM_PDF- Description: If set to
"True", the application will load existing redaction annotations from PDFs during the review step. - Default Value:
"True"
- Description: If set to
External Tool Paths
TESSERACT_FOLDER- Description: Path to the local Tesseract OCR installation folder.
- Default Value:
''
TESSERACT_DATA_FOLDER- Description: Path to the Tesseract trained data files (
tessdata). - Default Value:
"/usr/share/tessdata"
- Description: Path to the Tesseract trained data files (
POPPLER_FOLDER- Description: Path to the local Poppler installation’s
binfolder. - Default Value:
''
- Description: Path to the local Poppler installation’s
PADDLE_MODEL_PATH/SPACY_MODEL_PATH- Description: Custom directory for PaddleOCR and spaCy model storage, useful for environments like AWS Lambda.
- Default Value:
''(uses default location).
Local OCR (Tesseract & PaddleOCR)
CHOSEN_LOCAL_OCR_MODEL- Description: Choose the engine for local OCR:
"tesseract","paddle", or"hybrid". - Default Value:
"tesseract"
- Description: Choose the engine for local OCR:
SHOW_LOCAL_OCR_MODEL_OPTIONS- Description: If set to
"True", allows the user to select the local OCR model from the UI. - Default Value:
"False"
- Description: If set to
HYBRID_OCR_CONFIDENCE_THRESHOLD- Description: In “hybrid” mode, this is the Tesseract confidence score below which PaddleOCR will be used for re-extraction.
- Default Value:
65
HYBRID_OCR_PADDING- Description: In “hybrid” mode, padding added to the word’s bounding box before re-extraction.
- Default Value:
1
PADDLE_USE_TEXTLINE_ORIENTATION- Description: Toggles textline orientation detection for PaddleOCR.
- Default Value:
"False"
PADDLE_DET_DB_UNCLIP_RATIO- Description: Controls the expansion ratio of the detected text region in PaddleOCR.
- Default Value:
1.2
SAVE_EXAMPLE_TESSERACT_VS_PADDLE_IMAGES- Description: Saves comparison images when using “hybrid” OCR mode.
- Default Value:
"False"
SAVE_PADDLE_VISUALISATIONS- Description: Saves images with PaddleOCR’s detected bounding boxes overlaid.
- Default Value:
"False"
PREPROCESS_LOCAL_OCR_IMAGES- Description: If set to
"True", images will be preprocessed before local OCR. Can slow down processing. - Default Value:
"False"
- Description: If set to
Entity and Search Options
CHOSEN_COMPREHEND_ENTITIES/FULL_COMPREHEND_ENTITY_LIST- Description: The selected and available PII entity types for AWS Comprehend.
- Default Value: Predefined lists of entities (see script).
CHOSEN_REDACT_ENTITIES/FULL_ENTITY_LIST- Description: The selected and available PII entity types for the local model.
- Default Value: Predefined lists of entities (see script).
CUSTOM_ENTITIES- Description: A list of entities that are considered “custom” and may have special handling.
- Default Value:
['TITLES', 'UKPOSTCODE', 'STREETNAME', 'CUSTOM']
DEFAULT_SEARCH_QUERY- Description: The default text for the custom search/redact input box.
- Default Value:
''
DEFAULT_FUZZY_SPELLING_MISTAKES_NUM- Description: Default number of allowed spelling mistakes for fuzzy searches.
- Default Value:
1
DEFAULT_PAGE_MIN/DEFAULT_PAGE_MAX- Description: Default start and end pages for processing.
0for max means process all pages. - Default Value:
0for both.
- Description: Default start and end pages for processing.
Textract Feature Selection
DEFAULT_HANDWRITE_SIGNATURE_CHECKBOX- Description: The default options selected for Textract’s handwriting and signature detection.
- Default Value:
['Extract handwriting']
INCLUDE_FORM_EXTRACTION_TEXTRACT_OPTIONINCLUDE_LAYOUT_EXTRACTION_TEXTRACT_OPTIONINCLUDE_TABLE_EXTRACTION_TEXTRACT_OPTION- Description: Booleans (
"True"/"False") to include Forms, Layout, and Tables as selectable options for Textract analysis. - Default Value:
"False"for all.
- Description: Booleans (
Tabular Data Options
DO_INITIAL_TABULAR_DATA_CLEAN- Description: If
"True", performs an initial cleaning step on tabular data. - Default Value:
"True"
- Description: If
DEFAULT_TEXT_COLUMNS/DEFAULT_EXCEL_SHEETS- Description: Default values for specifying which columns or sheets to process in tabular files.
- Default Value:
[](empty list)
DEFAULT_TABULAR_ANONYMISATION_STRATEGY- Description: The default method for anonymizing tabular data (e.g., “redact completely”).
- Default Value:
"redact completely"
Language Options
Settings for multi-language support.
SHOW_LANGUAGE_SELECTION- Description: If set to
"True", a language selection dropdown will be visible in the UI. - Default Value:
"False"
- Description: If set to
DEFAULT_LANGUAGE_FULL_NAME/DEFAULT_LANGUAGE- Description: The default language’s full name (e.g., “english”) and its short code (e.g., “en”).
- Default Values:
"english","en"
textract_language_choices/aws_comprehend_language_choices- Description: Lists of supported language codes for Textract and Comprehend.
- Default Value:
['en', 'es', 'fr', 'de', 'it', 'pt']and['en', 'es']
MAPPED_LANGUAGE_CHOICES/LANGUAGE_CHOICES- Description: Paired lists of full language names and their corresponding short codes for the UI dropdown.
- Default Value: Predefined lists (see script).
Duplicate Detection Settings
DEFAULT_DUPLICATE_DETECTION_THRESHOLD- Description: The similarity score (0.0 to 1.0) above which documents/pages are considered duplicates.
- Default Value:
0.95
DEFAULT_MIN_CONSECUTIVE_PAGES- Description: Minimum number of consecutive pages that must be duplicates to be flagged.
- Default Value:
1
USE_GREEDY_DUPLICATE_DETECTION- Description: If
"True", uses a greedy algorithm that may find more duplicates but can be less precise. - Default Value:
"True"
- Description: If
DEFAULT_COMBINE_PAGES- Description: If
"True", text from the same page number across different files is combined before checking for duplicates. - Default Value:
"True"
- Description: If
DEFAULT_MIN_WORD_COUNT- Description: Pages with fewer words than this value will be ignored by the duplicate detector.
- Default Value:
10
REMOVE_DUPLICATE_ROWS- Description: If
"True", enables duplicate row detection in tabular data. - Default Value:
"False"
- Description: If
File Output Options
USE_GUI_BOX_COLOURS_FOR_OUTPUTS- Description: If
"True", the final redacted PDF will use the same redaction box colors as shown in the review UI. - Default Value:
"False"
- Description: If
CUSTOM_BOX_COLOUR- Description: Specifies the color for redaction boxes as an RGB tuple string, e.g.,
"(0, 0, 0)"for black. - Default Value:
"(0, 0, 0)"
- Description: Specifies the color for redaction boxes as an RGB tuple string, e.g.,
APPLY_REDACTIONS_IMAGES,APPLY_REDACTIONS_GRAPHICS,APPLY_REDACTIONS_TEXT- Description: Advanced control over how redactions are applied to underlying images, vector graphics, and text in the PDF, based on PyMuPDF options.
0is the default for a standard redaction workflow. - Default Value:
0for all.
- Description: Advanced control over how redactions are applied to underlying images, vector graphics, and text in the PDF, based on PyMuPDF options.
RETURN_PDF_FOR_REVIEW- Description: If set to
"True", a PDF with redaction boxes drawn on it (but text not removed) is generated for the “Review” tab. - Default Value:
"True"
- Description: If set to
RETURN_REDACTED_PDF- Description: If set to
'True', the application will return a fully redacted PDF at the end of the main task. - Default Value:
"True"
- Description: If set to
COMPRESS_REDACTED_PDF- Description: If set to
'True', the redacted PDF output will be compressed. - Default Value:
"False"
- Description: If set to
Direct Mode & Lambda Configuration
Settings for running the application from the command line (Direct Mode) or as an AWS Lambda function.
Direct Mode
RUN_DIRECT_MODE- Description: Set to
'True'to enable direct command-line mode. - Default Value:
'False'
- Description: Set to
DIRECT_MODE_DEFAULT_USER- Description: Default username for CLI requests.
- Default Value:
''
DIRECT_MODE_TASK- Description: The task to perform:
'redact'or'deduplicate'. - Default Value:
'redact'
- Description: The task to perform:
DIRECT_MODE_INPUT_FILE/DIRECT_MODE_OUTPUT_DIR- Description: Path to the input file and output directory for the task.
- Default Values:
'',output/
- Other
DIRECT_MODE_*variables:- Description: These variables allow for setting nearly all application options (e.g.,
DIRECT_MODE_PII_DETECTOR,DIRECT_MODE_SIMILARITY_THRESHOLD) directly for a single CLI run, overriding other configurations. - Default Value: Defaults are inherited from the main application settings (e.g.,
LOCAL_PII_OPTION,DEFAULT_DUPLICATE_DETECTION_THRESHOLD).
- Description: These variables allow for setting nearly all application options (e.g.,
Lambda Configuration
LAMBDA_POLL_INTERVAL- Description: Polling interval in seconds for checking Textract job status.
- Default Value:
30
LAMBDA_MAX_POLL_ATTEMPTS- Description: Maximum number of polling attempts before timeout.
- Default Value:
120
LAMBDA_PREPARE_IMAGES- Description: If
"True", prepares images for OCR processing within the Lambda environment. - Default Value:
"True"
- Description: If
LAMBDA_EXTRACT_SIGNATURES- Description: Enables signature extraction during Textract analysis in Lambda.
- Default Value:
"False"
LAMBDA_DEFAULT_USERNAME- Description: Default username for operations initiated by Lambda.
- Default Value:
"lambda_user"
Allow, Deny, & Whole Page Redaction Lists
GET_DEFAULT_ALLOW_LIST,GET_DEFAULT_DENY_LIST,GET_DEFAULT_WHOLE_PAGE_REDACTION_LIST- Description: Booleans (
"True"/"False") to enable the use of allow, deny, or whole-page redaction lists. - Default Value:
"False"
- Description: Booleans (
ALLOW_LIST_PATH,DENY_LIST_PATH,WHOLE_PAGE_REDACTION_LIST_PATH- Description: Local paths to the respective CSV list files.
- Default Value:
''
S3_ALLOW_LIST_PATH,S3_DENY_LIST_PATH,S3_WHOLE_PAGE_REDACTION_LIST_PATH- Description: Paths to the respective list files within the
DOCUMENT_REDACTION_BUCKET. - Default Value:
''
- Description: Paths to the respective list files within the
Cost Code Options
SHOW_COSTS- Description: If set to
'True', cost-related information will be displayed in the UI. - Default Value:
'False'
- Description: If set to
GET_COST_CODES- Description: Enables fetching and using cost codes. Set to
'True'to enable. - Default Value:
'False'
- Description: Enables fetching and using cost codes. Set to
DEFAULT_COST_CODE- Description: Specifies a default cost code.
- Default Value:
''
COST_CODES_PATH/S3_COST_CODES_PATH- Description: Local or S3 path to a CSV file containing available cost codes.
- Default Value:
''
ENFORCE_COST_CODES- Description: If set to
'True', makes the selection of a cost code mandatory. - Default Value:
'False'
- Description: If set to
Whole Document API Options (Textract Async)
SHOW_WHOLE_DOCUMENT_TEXTRACT_CALL_OPTIONS- Description: Controls whether UI options for asynchronous whole document Textract calls are displayed.
- Default Value:
'False'
TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_BUCKET- Description: The S3 bucket used for asynchronous Textract analysis.
- Default Value:
''
TEXTRACT_WHOLE_DOCUMENT_ANALYSIS_INPUT_SUBFOLDER/..._OUTPUT_SUBFOLDER- Description: Input and output subfolders within the analysis bucket.
- Default Values:
'input','output'
LOAD_PREVIOUS_TEXTRACT_JOBS_S3- Description: If set to
'True', the application will load data from previous Textract jobs stored in S3. - Default Value:
'False'
- Description: If set to
TEXTRACT_JOBS_S3_LOC/TEXTRACT_JOBS_S3_INPUT_LOC- Description: S3 subfolders where Textract job output and input are stored.
- Default Value:
'output','input'
TEXTRACT_JOBS_LOCAL_LOC- Description: The local subfolder for storing Textract job data.
- Default Value:
'output'
DAYS_TO_DISPLAY_WHOLE_DOCUMENT_JOBS- Description: Specifies the number of past days for which to display whole document Textract jobs.
- Default Value:
7