Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 1 | ========================== |
| 2 | Reference counting in pnfs |
Fred Isaman | 02c35fc | 2010-10-20 00:17:59 -0400 | [diff] [blame] | 3 | ========================== |
| 4 | |
| 5 | The are several inter-related caches. We have layouts which can |
| 6 | reference multiple devices, each of which can reference multiple data servers. |
| 7 | Each data server can be referenced by multiple devices. Each device |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 8 | can be referenced by multiple layouts. To keep all of this straight, |
Fred Isaman | 02c35fc | 2010-10-20 00:17:59 -0400 | [diff] [blame] | 9 | we need to reference count. |
| 10 | |
| 11 | |
| 12 | struct pnfs_layout_hdr |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 13 | ====================== |
| 14 | |
Fred Isaman | 02c35fc | 2010-10-20 00:17:59 -0400 | [diff] [blame] | 15 | The on-the-wire command LAYOUTGET corresponds to struct |
| 16 | pnfs_layout_segment, usually referred to by the variable name lseg. |
Masanari Iida | c9f3f2d | 2013-07-18 01:29:12 +0900 | [diff] [blame] | 17 | Each nfs_inode may hold a pointer to a cache of these layout |
Fred Isaman | 02c35fc | 2010-10-20 00:17:59 -0400 | [diff] [blame] | 18 | segments in nfsi->layout, of type struct pnfs_layout_hdr. |
| 19 | |
| 20 | We reference the header for the inode pointing to it, across each |
| 21 | outstanding RPC call that references it (LAYOUTGET, LAYOUTRETURN, |
| 22 | LAYOUTCOMMIT), and for each lseg held within. |
| 23 | |
| 24 | Each header is also (when non-empty) put on a list associated with |
| 25 | struct nfs_client (cl_layouts). Being put on this list does not bump |
| 26 | the reference count, as the layout is kept around by the lseg that |
| 27 | keeps it in the list. |
| 28 | |
| 29 | deviceid_cache |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 30 | ============== |
| 31 | |
Fred Isaman | 02c35fc | 2010-10-20 00:17:59 -0400 | [diff] [blame] | 32 | lsegs reference device ids, which are resolved per nfs_client and |
| 33 | layout driver type. The device ids are held in a RCU cache (struct |
| 34 | nfs4_deviceid_cache). The cache itself is referenced across each |
| 35 | mount. The entries (struct nfs4_deviceid) themselves are held across |
| 36 | the lifetime of each lseg referencing them. |
| 37 | |
| 38 | RCU is used because the deviceid is basically a write once, read many |
| 39 | data structure. The hlist size of 32 buckets needs better |
| 40 | justification, but seems reasonable given that we can have multiple |
| 41 | deviceid's per filesystem, and multiple filesystems per nfs_client. |
| 42 | |
| 43 | The hash code is copied from the nfsd code base. A discussion of |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 44 | hashing and variations of this algorithm can be found `here. |
| 45 | <http://groups.google.com/group/comp.lang.c/browse_thread/thread/9522965e2b8d3809>`_ |
Fred Isaman | 02c35fc | 2010-10-20 00:17:59 -0400 | [diff] [blame] | 46 | |
| 47 | data server cache |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 48 | ================= |
| 49 | |
Fred Isaman | 02c35fc | 2010-10-20 00:17:59 -0400 | [diff] [blame] | 50 | file driver devices refer to data servers, which are kept in a module |
| 51 | level cache. Its reference is held over the lifetime of the deviceid |
| 52 | pointing to it. |
Fred Isaman | 80fe2b1 | 2011-03-01 01:34:23 +0000 | [diff] [blame] | 53 | |
| 54 | lseg |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 55 | ==== |
| 56 | |
Fred Isaman | 80fe2b1 | 2011-03-01 01:34:23 +0000 | [diff] [blame] | 57 | lseg maintains an extra reference corresponding to the NFS_LSEG_VALID |
| 58 | bit which holds it in the pnfs_layout_hdr's list. When the final lseg |
| 59 | is removed from the pnfs_layout_hdr's list, the NFS_LAYOUT_DESTROYED |
| 60 | bit is set, preventing any new lsegs from being added. |
Sachin Bhamare | 18d98f6 | 2012-03-19 20:47:58 -0700 | [diff] [blame] | 61 | |
| 62 | layout drivers |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 63 | ============== |
Sachin Bhamare | 18d98f6 | 2012-03-19 20:47:58 -0700 | [diff] [blame] | 64 | |
Tom Haynes | 8f9cdcb | 2015-01-12 11:51:45 -0800 | [diff] [blame] | 65 | PNFS utilizes what is called layout drivers. The STD defines 4 basic |
| 66 | layout types: "files", "objects", "blocks", and "flexfiles". For each |
| 67 | of these types there is a layout-driver with a common function-vectors |
| 68 | table which are called by the nfs-client pnfs-core to implement the |
| 69 | different layout types. |
Sachin Bhamare | 18d98f6 | 2012-03-19 20:47:58 -0700 | [diff] [blame] | 70 | |
Tom Haynes | 8f9cdcb | 2015-01-12 11:51:45 -0800 | [diff] [blame] | 71 | Files-layout-driver code is in: fs/nfs/filelayout/.. directory |
Masanari Iida | 0d6f3eb | 2016-02-18 12:26:13 +0900 | [diff] [blame] | 72 | Blocks-layout-driver code is in: fs/nfs/blocklayout/.. directory |
Tom Haynes | 8f9cdcb | 2015-01-12 11:51:45 -0800 | [diff] [blame] | 73 | Flexfiles-layout-driver code is in: fs/nfs/flexfilelayout/.. directory |
Sachin Bhamare | 18d98f6 | 2012-03-19 20:47:58 -0700 | [diff] [blame] | 74 | |
Sachin Bhamare | 18d98f6 | 2012-03-19 20:47:58 -0700 | [diff] [blame] | 75 | blocks-layout setup |
Daniel W. S. Almeida | 34e75cf | 2020-01-29 01:49:13 -0300 | [diff] [blame] | 76 | =================== |
Sachin Bhamare | 18d98f6 | 2012-03-19 20:47:58 -0700 | [diff] [blame] | 77 | |
| 78 | TODO: Document the setup needs of the blocks layout driver |