blob: 7c470ecdc3a94b973f1fde2c9448de0f4c7a9fa9 [file] [log] [blame]
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -03001==========================
2Reference counting in pnfs
Fred Isaman02c35fc2010-10-20 00:17:59 -04003==========================
4
5The are several inter-related caches. We have layouts which can
6reference multiple devices, each of which can reference multiple data servers.
7Each data server can be referenced by multiple devices. Each device
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -03008can be referenced by multiple layouts. To keep all of this straight,
Fred Isaman02c35fc2010-10-20 00:17:59 -04009we need to reference count.
10
11
12struct pnfs_layout_hdr
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -030013======================
14
Fred Isaman02c35fc2010-10-20 00:17:59 -040015The on-the-wire command LAYOUTGET corresponds to struct
16pnfs_layout_segment, usually referred to by the variable name lseg.
Masanari Iidac9f3f2d2013-07-18 01:29:12 +090017Each nfs_inode may hold a pointer to a cache of these layout
Fred Isaman02c35fc2010-10-20 00:17:59 -040018segments in nfsi->layout, of type struct pnfs_layout_hdr.
19
20We reference the header for the inode pointing to it, across each
21outstanding RPC call that references it (LAYOUTGET, LAYOUTRETURN,
22LAYOUTCOMMIT), and for each lseg held within.
23
24Each header is also (when non-empty) put on a list associated with
25struct nfs_client (cl_layouts). Being put on this list does not bump
26the reference count, as the layout is kept around by the lseg that
27keeps it in the list.
28
29deviceid_cache
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -030030==============
31
Fred Isaman02c35fc2010-10-20 00:17:59 -040032lsegs reference device ids, which are resolved per nfs_client and
33layout driver type. The device ids are held in a RCU cache (struct
34nfs4_deviceid_cache). The cache itself is referenced across each
35mount. The entries (struct nfs4_deviceid) themselves are held across
36the lifetime of each lseg referencing them.
37
38RCU is used because the deviceid is basically a write once, read many
39data structure. The hlist size of 32 buckets needs better
40justification, but seems reasonable given that we can have multiple
41deviceid's per filesystem, and multiple filesystems per nfs_client.
42
43The hash code is copied from the nfsd code base. A discussion of
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -030044hashing and variations of this algorithm can be found `here.
45<http://groups.google.com/group/comp.lang.c/browse_thread/thread/9522965e2b8d3809>`_
Fred Isaman02c35fc2010-10-20 00:17:59 -040046
47data server cache
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -030048=================
49
Fred Isaman02c35fc2010-10-20 00:17:59 -040050file driver devices refer to data servers, which are kept in a module
51level cache. Its reference is held over the lifetime of the deviceid
52pointing to it.
Fred Isaman80fe2b12011-03-01 01:34:23 +000053
54lseg
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -030055====
56
Fred Isaman80fe2b12011-03-01 01:34:23 +000057lseg maintains an extra reference corresponding to the NFS_LSEG_VALID
58bit which holds it in the pnfs_layout_hdr's list. When the final lseg
59is removed from the pnfs_layout_hdr's list, the NFS_LAYOUT_DESTROYED
60bit is set, preventing any new lsegs from being added.
Sachin Bhamare18d98f62012-03-19 20:47:58 -070061
62layout drivers
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -030063==============
Sachin Bhamare18d98f62012-03-19 20:47:58 -070064
Tom Haynes8f9cdcb2015-01-12 11:51:45 -080065PNFS utilizes what is called layout drivers. The STD defines 4 basic
66layout types: "files", "objects", "blocks", and "flexfiles". For each
67of these types there is a layout-driver with a common function-vectors
68table which are called by the nfs-client pnfs-core to implement the
69different layout types.
Sachin Bhamare18d98f62012-03-19 20:47:58 -070070
Tom Haynes8f9cdcb2015-01-12 11:51:45 -080071Files-layout-driver code is in: fs/nfs/filelayout/.. directory
Masanari Iida0d6f3eb2016-02-18 12:26:13 +090072Blocks-layout-driver code is in: fs/nfs/blocklayout/.. directory
Tom Haynes8f9cdcb2015-01-12 11:51:45 -080073Flexfiles-layout-driver code is in: fs/nfs/flexfilelayout/.. directory
Sachin Bhamare18d98f62012-03-19 20:47:58 -070074
Sachin Bhamare18d98f62012-03-19 20:47:58 -070075blocks-layout setup
Daniel W. S. Almeida34e75cf2020-01-29 01:49:13 -030076===================
Sachin Bhamare18d98f62012-03-19 20:47:58 -070077
78TODO: Document the setup needs of the blocks layout driver