block: add BIO_COMPLETE_IN_TASK for task-context completion#739
Open
blktests-ci[bot] wants to merge 8 commits intolinus-master_basefrom
Open
block: add BIO_COMPLETE_IN_TASK for task-context completion#739blktests-ci[bot] wants to merge 8 commits intolinus-master_basefrom
blktests-ci[bot] wants to merge 8 commits intolinus-master_basefrom
Conversation
Some bio completion handlers need to run in task context but bio_endio() can be called from IRQ context (e.g. buffer_head writeback). Add a BIO_COMPLETE_IN_TASK flag that bio submitters can set to request task-context completion of their bi_end_io callback. When bio_endio() sees this flag and is running in non-task context, it queues the bio to a per-cpu lockless list and schedules a delayed work item to call bi_end_io() from task context. The delayed work uses a 1-jiffie delay to allow batches of completions to accumulate before processing. A CPU hotplug dead callback drains any remaining bios from the departing CPU's batch. This will be used to enable RWF_DONTCACHE for block devices, and could be used for other subsystems like fscrypt that need task-context bio completion. Suggested-by: Matthew Wilcox <willy@infradead.org> Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Set BIO_COMPLETE_IN_TASK on iomap writeback bios when a dropbehind folio is added. This ensures that bi_end_io runs in task context, where folio_end_dropbehind() can safely invalidate folios. With the bio layer now handling task-context deferral generically, IOMAP_IOEND_DONTCACHE is no longer needed, as XFS no longer needs to route DONTCACHE ioends through its completion workqueue. Remove the flag and its NOMERGE entry. Without the NOMERGE, regular I/Os that get merged with a dropbehind folio will also have their completion deferred to task context. Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Block device buffered reads and writes already pass through filemap_read() and iomap_file_buffered_write() respectively, both of which handle IOCB_DONTCACHE. Enable RWF_DONTCACHE for block device files by setting FOP_DONTCACHE in def_blk_fops. For CONFIG_BUFFER_HEAD=y paths, add block_write_begin_iocb() which threads the kiocb through so that buffer_head-based I/O can use DONTCACHE behavior. The existing block_write_begin() is preserved as a wrapper that passes a NULL iocb. Set BIO_COMPLETE_IN_TASK in submit_bh_wbc() when the folio has dropbehind so that buffer_head writeback completions get deferred to task context. CONFIG_BUFFER_HEAD=n paths are handled by the previously added iomap BIO_COMPLETE_IN_TASK support. This support is useful for databases that operate on raw block devices, among other userspace applications. Signed-off-by: Tal Zussman <tz2294@columbia.edu>
Replace the bio-flag based interface with an explicit bio_complete_in_task() API. The advantage is that this can also be called from inside the ->bi_end_io callback and thus dynamically. This will be important to use it for fserror reporting. Signed-off-by: Christoph Hellwig <hch@lst.de>
As described in commit c99fab6, some block drivers might call into ->bi_end_io from non-preemptible context. Copy and past the logic from that commit, although having a core helper for it would be nicer. Signed-off-by: Christoph Hellwig <hch@lst.de>
Replace out own hand-crafted complete in task context scheme with the generic block code. Signed-off-by: Christoph Hellwig <hch@lst.de>
Replace out own hand-crafted complete in task context scheme with the generic block code. Signed-off-by: Christoph Hellwig <hch@lst.de>
Commit 3fffb58 ("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of a kthread_work. Signed-off-by: Christoph Hellwig <hch@lst.de>
Author
|
Upstream branch: d60bc14 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request for series with
subject: block: add BIO_COMPLETE_IN_TASK for task-context completion
version: 1
url: https://patchwork.kernel.org/project/linux-block/list/?series=1079338