Added config file support (.docfd-config)
--list-scripts flag--script and --start-with-script to look up script in data directory if provided path is a file name and not present in current directory, appending .docfd-script if needed during lookup in script directoryOther minor fixes aiming to further reduce crashes
sqlite3 >= 5.4.2 for a crash fixSQLite connection code cleanup
v has been used to hide itRemoved fzf dependency entirely
Command history memory optimisation
Ctrl+X under SCRIPTSy key binding to copy in LINKS? to < and >Changed to using left right arrows to adjust screen split
Tab/Shift+Tab no longer control screen splitv which hides the bottom right pane?Esc as alternative to Enter to SEARCH and FILTER for confirming inputMade the UX of various input fields slightly more polished
Esc for cancelling during SAVE-SCRIPTFixed file scanning logic when hidden paths are directly specified or current working directory is a hidden path
--hidden is also specified, which was counterintuitivetimedesc >= 3.1.2, which addresses the issueFixed missing stop signal passing for content:"..." filter expression handling
content:"..." were not cancelled properlyFixed key binding info grid
l key binding infoNote: This update contains a feature that requires removing the existing index DB to take effect.
Docfd script comment support improvement
; prefix for system comments, which are not preserved after editing of command history# now denotes user comment and is preserved after editing of command history# is also preserved during saving a session as scriptAdded link opening support via LINKS mode
l opens LINKS mode with the same navigation keybinds
Enter to open linko to open link and remain in LINKSxh to clear command history quicklyInternal refactoring
Moved screen split handling to the level of session state instead of plain UI state
Shift+Tab for changing screen split ratio in the other direction, and removed the "rotating" behaviour of TabMoved sorting handling to the level of document store command instead of just plain UI update
Fixed default cache directory location on macOS
~/Library/Application Support to ~/Library/Caches--reverse to fzf invocation for better UXCtrl+D for script deletionThis contains a breaking DB change, you will need to remove index DB generated by Docfd version prior to 12.0.0-alpha.13
Moved to using a global word table to reduce index DB size and speed up search (12.0.0-alpha.13)
Added further search speed optimizations (12.0.0)
Added a first word candidate pruning stage based on length of the first search word
Added content view pane scrolling (12.0.0-alpha.5, 12.0.0-alpha.8)
-/=Key binding info grid improvements (12.0.0-alpha.4)
--paths-from - to accept list of paths from stdin (12.0.0-alpha.3)Added more marking key bindings (12.0.0-alpha.4)
mark listed (ml) marks all currently listed documentsunmark listed (Ml) unmarks all currently listed documents--open-with placeholder handling fixes (12.0.0-alpha.4)
{page_num} and {line_num} crashes in 11.0.1 when there are no search resultsAdded sorting to document list (12.0.0-alpha.11, 12.0.0)
s for sort ascending mode and Shift+S for sort descending modeUnder the sort modes, the sort by types are as follows:
p sort by pathd sort by path dates sort by scorem sort by modification timef sort by an interactive fzf search
Adjusted attributes listed in document list entry (12.0.0-alpha.11)
Added further search speed optimizations:
Added a first word candidate pruning stage based on length of the first search word
Unix.waitpid on macOS compared to LinuxFixed document sorting fallback behavior
--sort-no-score--open-with to accept a list of extensions, e.g. --open-with ts,js:detached="... {path}"Added sort by fzf functionality
Under sort mode
f sort by an interactive fzf search
Moved to using a global word table to reduce index DB size and speed up search
Added more path date extraction formats
yyyy-mmm-dd, yyyy-mmmm-dd, dd-mmm-yyyy, dd-mmmm-yyyy- is an optional separator that is not a digit and not a letterMade resetting of search result selection and content view offset less aggressive
d when searching or filtering is ongoingSwapped all mutexes to Eio mutexes to hopefully remove the very random freezes that occur quite rarely
Added sorting to document list
s for sort ascending mode and Shift+S for sort descending modeUnder the sort modes, the sort by types are as follows:
p sort by pathd sort by path dates sort by scorem sort by modification timeAdded --sort and --sort-no-score
--files-without-match is usedyyyymmdd path date extractionmod-date to filter languageAdjusted attributes listed in document list entry
Reworked --script into --script and --start-with-script
--script is now only for non-interactive use--start-with-script is only for interactive use--filter vs --start-with-filter and --search vs --start-with-searchReworked the internal architecture of document store snapshots storage and management
Document_store_manager, along with improvements to snapshot handling logic in generalTab handling in edit fields to reduce friction in UXnano-style autocomplete to save commands field with listing of existing scripts--commands-from to --scriptImproved content view pane scrolling control
--commands-frommark listed and unmark listed to template command history file help infoFixed not operator parsing
not ext:txt and not ext:md would be parsed as not (ext:txt and not ext:md), which is not what is typically expectednot now binds tightly, so not ext:txt and not ext:md is parsed as (not ext:txt) and (not ext:md)Added content view pane scrolling
-/=Removed extraneous marking functionality
mark unlistedunmark unlistedAdded "..." as a shorthand to content:"..." to filter expression
content:keyword AND path-date:>2025-01-01 can be written as "keyword" AND path-date:>2025-01-01pathfuzzy:... would be parsed as content queries insteadAdded additional marking functionality
mark listed (ml) marks all currently listed documentsmark unlisted (mL) marks all currently unlisted documentsunmark listed (Ml) unmarks all currently listed documentsunmark unlisted (ML) unmarks all currently unlisted documentsunmark all is moved to key binding MaTab key to key binding info gridMinor fix for {line_num} placeholder handling in --open-with
{line_num} is specified in --open-with and user opens a text file when no search has been made{line_num} to 1 when there are no search results presentMinor fix for {page_num} and {search_word} placeholders handling in --open-with
{page_num} or {search_word} is specified in --open-with and user opens a PDF file when no search has been made{page_num} to 1 and {search_word} to empty string when there are no search results presentAdjusted SQL indices and swapped to specialized SQL queries for exact and prefix search terms, e.g. 'hello, ^worl
Fixed command history recomputation not using the reloaded version of document store
Enter in Docfd (after which Docfd reloads the file for you), and you hit h to modify the command historyAdded missing SQL transaction in code path for reloading a single document
Enter in Docfd, unless the text file was very smallUpdated --paths-from argument handling
--paths-from - for accepting list of paths from stdin--paths-from path-list0.txt,path-list1.txtRemoved builtin piping to fzf triggered by providing ? as a file path, e.g. docfd ?
--paths-from - handling makes this obsolete and a lot less flexible by comparisonFixed interaction between search and filter
Starting a new filtering operation also still cancels any ongoing search. This is fine since the search results are refreshed after the filtering has been completed.
The refreshing of the search results also means that the following sequences of events are still handled correctly, namely they still arrive at the same normal form of the document store:
Example 1:
f_exp0 (filtering is canceled by step (2), but the updating of filter expression is never canceled)s_exp0 (search is canceled by step (2), but the updating of search expression is never canceled)f_exp1 (refreshes search results using s_exp0)Example 2:
s_exp0 (search is canceled by step (1), but updating of search expression is never canceled)f_exp0 (this stage is canceled by step (2), either during the filtering or during the refreshing of search results, but the updating of filter expression is never canceled)f_exp1 (refreshes search results using s_exp0)Fixed concurrency issue where an update of document store may cause the filter field and search field in UI to be out of sync with the actual filter expression and search expression used by the underlying document store
Suppose we have the following sequence of events:
store0 carries filter expression f_exp0 and search expression s_exp0, which we write as pair (f_exp0, s_exp0)(1) User initiates filter/search operation by placing (f_exp1, s_exp1) into the input fields.
We name the document store resulting from this filter/search operation as store1a, which carries (f_exp1, s_exp1) when finalized.
(2) While filter/search operation is ongoing, user drops a set of documents from the current document store. Since store1a is not finalized yet, the current document store is still store0, thus the new document store encoding the result of the drop operation, store1b, is computed from store0 instead of store1a.
In other words, both store1a and store1b share store0 as their parent. Note that store1b carries (f_exp0, s_exp0) as inherited from store0, since a drop operation does not alter the filter expression or search expression.
store1a, and instead places store1b as the current document store.However, this means the input fields are (f_exp1, s_exp1) while the current document store store1b actually carries (f_exp0, s_exp0).
The fix in this update is then to add an extra "sync from input fields" step whenever a document store is updated. To illustrate, we continue from the above sequence of events, where the updated version of Docfd carries out the following step missing from previous versions.
(f_exp0, s_exp0)In practice this is very unlikely to occur with human input, as the modes that update document store are disabled if document store manager is carrying out any ongoing filtering or search.
However, since the UI is async, there will be gaps in timing between UI input/feedback and actual updates of values, opening up to TOCTOU problems. So there is always a chance that a document store update will be requested before the modes are are disabled.
Added path-date clause to query expression
path-date:>=2025-01-01 AND path-date:<2025-02-01 would allow /home/user/meeting-notes-2025-01-10.md to pass throughAdded a more powerful filter mode that replaces the filter glob mode and "pipe to fzf" feature
fq exit key binding to avoid accidental exitingMinor fix for search scope narrowing logic:
Added missing commands in the list of possible commands in the command history file template
clear searchclear filterMinor breaking change, filter regex mode should have been called filter glob mode
fr is changed to fgAdded --open-with to allow customising the command used to open a file based on file extension
--open-with pdf:detached='okular {path}'Added non-interactive use of --commands-from
--commands-from with -l/--files-with-matchAdjustments to search scope narrowing
narrow level: 0 for resetting the search scopes of all documents back to fullReworked search into multi-stage pipeline
Minor fixes
file.txt in docfd file.txt. This means "reload all documents" now does not error out due to files becoming no longer accessible.Minor fix for "reload all doucments" when fzf was used to pick documents initially, i.e. docfd [PATH]... ?, or any variation where ? appears anywhere in the path list
docfd ? alike is used, "reload all" no longer discovers new filesMinor fix for "filter files via fzf" functionality
Ctrl+C, Ctrl+Q), Docfd also closes with itAdded back index DB entry pruning
--cache-soft-limit to --cache-limit to reflect the new pruning logicdoc_id allocation strategy to minimise doc_id size in DBd is pressedReworked document indexing into a multi-stage pipeline
Breaking changes in index DB design - index DBs made by previous version of Docfd are not compatible
Added functionality to filter files via fzf
f enters filter mode
f again activates filter files via fzf functionalityr activates the filter regex mode, which was previously just called the filter modeFixed incomplete search results when file path filter field is updated while search is ongoing
Removed --no-cache flag
Swapped to using proper unicode segmentation for tokenisation
Minor UI/UX fixes
Added missing update of search and filter status when undoing/redoing, or when replaying command history
Swapped over to using SQLite for index
Memory usage is much slimmer/stays flat
Added token length limit of 500 bytes to accommodate word table limit in index DB
Ctrl+C exit key binding to key binding info on screenUpdated exit keys
q, Ctrl+Q or Ctrl+CEscNow defaults to not scanning hidden files and directories
--hidden flag--add-exts and --single-line-add-exts to be specified multiple times--sample or --searchAdded --no-pdftotext and --no-pandoc flags
drop path command to just dropdrop all exceptChanged default cache size from 100 to 10000
--cache-size to --cache-soft-limith to ?--commands-from command line argument--tokens-per-search-scope-level command line argumentConcurrency related bug fixes
--index-chunk-token-count to --index-chunk-size--sample-count-per-doc to --samples-per-docReworked asynchronous search/filter UI code to avoid noticeable lag due to waiting for cancellations that take too long
--markdown-headings atx from pandoc commandline argumentsAlt+U undo key bindingAlt+E redo key bindingCtrl+Q exit key bindingAdded initial macOS support
Major speedup from letting pdftotext output everything in one pass and split on Docfd side instead of asking pdftotext to output one page per invocation
pdftotext filters the form feed character from content), but should be rareg and G key bindings for going to top and bottom of document list respectively-l/--files-with-match and --files-without-match for printing just paths in non-interactive modeSlimmed down memory usage by switching to OCaml 5.2 which enables use of Gc.compact
h key binding to rotate key binding info grid~ to match spacesFixed random UI freezes when updating search field
--max-linked-token-search-distAdded smart additional line grabbing for search result printing
--search-result-print-snippet-min-size N
N non-space tokens, then Docfd tries to add surrounding lines to the snippet to give better context.--search-result-print-snippet-max-add-lines
--search to show all search results--sample that uses --search previous behavior where (by default) only a handful of top search results are picked for each document--search-result-count-per-doc to --sample-count-per-docAdded --color and --underline for controlling behavior of search result printing, they can take one of:
neveralwaysautoTab key presses--max-token-search-distTab key presses--glob argumentAdded single line search mode arguments
--single-line-exts--single-line-add-exts--single-line-glob--single-lineFixed crash on empty file
max_line_num in Content_and_search_result_render.content_snippetShift+P and p within TUI to not exit after printing, allowing printing of more results--paths-from to be specified multiple timesFixed handling of .htm files
htm is not a valid value for pandoc's --format argumenthtml before being passed to pandocChanged --max-depth:
Added status to search bar:
OK means Docfd is idling... means Docfd is searchingERR means Docfd failed to parse the search expressionAdded dynamic search distance adjustment based on notion of linked tokens
- and > are linked in ->, but not in - >Replaced word with token in the following options for consistency
--max-word-search-dist--index-chunk-word-countword with token in user-facing textFixed crash from search result snippet being bigger the content view pane
Content_and_search_result_render.color_word_image_gridAdded key bindings
p: exit and print search result to stderrShift+P: exit and print file path to stderr--debug-log - to use stderr instead of stdoutAdded non-interactive search mode where search results are printed to stdout
--search EXP invokes non-interactive search mode with search expression EXP--search-result-count-per-document sets the number of top search results printed per document--search-result-print-text-width sets the text width to use when printing--start-with-search to prefill the search field in interactive modeRemoved content requirement expression from multi-file view
Added word based line wrapping to following components of document list in multi-file view
--paths-from to specify a file containing list of paths to (also) be scanned--max-fuzzy-edit to --max-fuzzy-edit-dist for consistencyChanged the logic of determining when to use stdin as document source
Restored behaviour of skipping file extension checks for top-level user specified files. This behaviour was likely removed during some previous overhaul.
docfd bin/docfd.ml will now open the file just fine without --add-exts mlAdded consideration for balanced opening closing symbols in search result ranking
(), [], {}Fixed crash from reading from stdin
Notty_unix.Term.release after closing the underlying file descriptor in stdin input mode? in search expressionNotty_unix.Term.t when neededEsc, Ctrl+C, and Ctrl+QfzfAdded PDF viewer integration for:
Notty_unix.Term.tXDG_CACHE_HOME/docfd, which overall defaults to $HOME/.cache/docfdAdded cache related arguments
--cache-dir--cache-size--no-cache--debug to --debug-log to support outputting debug log to a fileFixed file opening failure due to exhausting file descriptors
Document.of_path in Eio.Fiber.List.filter_map--index-only flagFixed crash from using mouse scrolling in multi-file view
--add-exts argument for additional file extensionsTuned search procedure and search result ranking
VISUAL and EDITOR to program startPerformance tuning
Performance tuning
Performance tuning
max-word-search-range to max-word-search-distAdded code to open selected text file to selected search result for:
--exts argument for configuring file extensions recognizedpdftotextFixed crashing and incorrect rendering in some cases of files with blank lines
Index.line_count being incorrectly calculatedAdded auto refresh on change of file
r keyBumped the default word search range from 15 to 40
Nottui_unix term creation so pre TUI printing like --version would workOptimized overall memory footprint
Added limit to word search range of each step in content search
--max-depth option to limit scanning depth--tags and --ltags output slightlyUpgraded --tags and --ltags output to be more human readable when output is terminal
Added support for alternative tag section syntax
| ... |@ ... @--tags and --ltags flags-s for case-insensitive substring tag match-p to -e for exact tag match