Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 1 | .. _slub: |
| 2 | |
| 3 | ========================== |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 4 | Short users guide for SLUB |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 5 | ========================== |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 6 | |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 7 | The basic philosophy of SLUB is very different from SLAB. SLAB |
| 8 | requires rebuilding the kernel to activate debug options for all |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 9 | slab caches. SLUB always includes full debugging but it is off by default. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 10 | SLUB can enable debugging only for selected slabs in order to avoid |
| 11 | an impact on overall system performance which may make a bug more |
| 12 | difficult to find. |
| 13 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 14 | In order to switch debugging on one can add an option ``slub_debug`` |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 15 | to the kernel command line. That will enable full debugging for |
| 16 | all slabs. |
| 17 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 18 | Typically one would then use the ``slabinfo`` command to get statistical |
| 19 | data and perform operation on the slabs. By default ``slabinfo`` only lists |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 20 | slabs that have data in them. See "slabinfo -h" for more options when |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 21 | running the command. ``slabinfo`` can be compiled with |
| 22 | :: |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 23 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 24 | gcc -o slabinfo tools/vm/slabinfo.c |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 25 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 26 | Some of the modes of operation of ``slabinfo`` require that slub debugging |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 27 | be enabled on the command line. F.e. no tracking information will be |
| 28 | available without debugging on and validation can only partially |
| 29 | be performed if debugging was not switched on. |
| 30 | |
| 31 | Some more sophisticated uses of slub_debug: |
| 32 | ------------------------------------------- |
| 33 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 34 | Parameters may be given to ``slub_debug``. If none is specified then full |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 35 | debugging is enabled. Format: |
| 36 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 37 | slub_debug=<Debug-Options> |
| 38 | Enable options for all slabs |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 39 | |
Aaron Tomlin | c5fd3ca | 2018-10-26 15:03:15 -0700 | [diff] [blame] | 40 | slub_debug=<Debug-Options>,<slab name1>,<slab name2>,... |
| 41 | Enable options only for select slabs (no spaces |
| 42 | after a comma) |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 43 | |
Vlastimil Babka | e17f1df | 2020-08-06 23:18:35 -0700 | [diff] [blame] | 44 | Multiple blocks of options for all slabs or selected slabs can be given, with |
| 45 | blocks of options delimited by ';'. The last of "all slabs" blocks is applied |
| 46 | to all slabs except those that match one of the "select slabs" block. Options |
| 47 | of the first "select slabs" blocks that matches the slab's name are applied. |
| 48 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 49 | Possible debug options are:: |
| 50 | |
Laura Abbott | becfda6 | 2016-03-15 14:55:06 -0700 | [diff] [blame] | 51 | F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS |
| 52 | Sorry SLAB legacy issues) |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 53 | Z Red zoning |
| 54 | P Poisoning (object and padding) |
| 55 | U User tracking (free and alloc) |
| 56 | T Trace (please only use on single slabs) |
Andrew Morton | a3df692 | 2020-06-01 21:46:00 -0700 | [diff] [blame] | 57 | A Enable failslab filter mark for the cache |
David Rientjes | fa5ec8a | 2009-07-07 00:14:14 -0700 | [diff] [blame] | 58 | O Switch debugging off for caches that would have |
| 59 | caused higher minimum slab orders |
Christoph Lameter | f0630ff | 2007-07-15 23:38:14 -0700 | [diff] [blame] | 60 | - Switch all debugging off (useful if the kernel is |
| 61 | configured with CONFIG_SLUB_DEBUG_ON) |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 62 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 63 | F.e. in order to boot just with sanity checks and red zoning one would specify:: |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 64 | |
| 65 | slub_debug=FZ |
| 66 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 67 | Trying to find an issue in the dentry cache? Try:: |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 68 | |
Itaru Kitayama | 989a724 | 2008-03-05 15:07:30 -0800 | [diff] [blame] | 69 | slub_debug=,dentry |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 70 | |
Aaron Tomlin | c5fd3ca | 2018-10-26 15:03:15 -0700 | [diff] [blame] | 71 | to only enable debugging on the dentry cache. You may use an asterisk at the |
| 72 | end of the slab name, in order to cover all slabs with the same prefix. For |
| 73 | example, here's how you can poison the dentry cache as well as all kmalloc |
Tobin C. Harding | 11ede50 | 2019-01-31 15:06:22 +1100 | [diff] [blame] | 74 | slabs:: |
Aaron Tomlin | c5fd3ca | 2018-10-26 15:03:15 -0700 | [diff] [blame] | 75 | |
| 76 | slub_debug=P,kmalloc-*,dentry |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 77 | |
| 78 | Red zoning and tracking may realign the slab. We can just apply sanity checks |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 79 | to the dentry cache with:: |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 80 | |
Itaru Kitayama | 989a724 | 2008-03-05 15:07:30 -0800 | [diff] [blame] | 81 | slub_debug=F,dentry |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 82 | |
David Rientjes | fa5ec8a | 2009-07-07 00:14:14 -0700 | [diff] [blame] | 83 | Debugging options may require the minimum possible slab order to increase as |
| 84 | a result of storing the metadata (for example, caches with PAGE_SIZE object |
| 85 | sizes). This has a higher liklihood of resulting in slab allocation errors |
| 86 | in low memory situations or if there's high fragmentation of memory. To |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 87 | switch off debugging for such caches by default, use:: |
David Rientjes | fa5ec8a | 2009-07-07 00:14:14 -0700 | [diff] [blame] | 88 | |
| 89 | slub_debug=O |
| 90 | |
Vlastimil Babka | e17f1df | 2020-08-06 23:18:35 -0700 | [diff] [blame] | 91 | You can apply different options to different list of slab names, using blocks |
| 92 | of options. This will enable red zoning for dentry and user tracking for |
| 93 | kmalloc. All other slabs will not get any debugging enabled:: |
| 94 | |
| 95 | slub_debug=Z,dentry;U,kmalloc-* |
| 96 | |
| 97 | You can also enable options (e.g. sanity checks and poisoning) for all caches |
| 98 | except some that are deemed too performance critical and don't need to be |
| 99 | debugged by specifying global debug options followed by a list of slab names |
| 100 | with "-" as options:: |
| 101 | |
| 102 | slub_debug=FZ;-,zs_handle,zspage |
| 103 | |
Vlastimil Babka | ad38b5b | 2020-08-06 23:18:38 -0700 | [diff] [blame] | 104 | The state of each debug option for a slab can be found in the respective files |
| 105 | under:: |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 106 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 107 | /sys/kernel/slab/<slab name>/ |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 108 | |
Vlastimil Babka | ad38b5b | 2020-08-06 23:18:38 -0700 | [diff] [blame] | 109 | If the file contains 1, the option is enabled, 0 means disabled. The debug |
| 110 | options from the ``slub_debug`` parameter translate to the following files:: |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 111 | |
Vlastimil Babka | ad38b5b | 2020-08-06 23:18:38 -0700 | [diff] [blame] | 112 | F sanity_checks |
| 113 | Z red_zone |
| 114 | P poison |
| 115 | U store_user |
| 116 | T trace |
| 117 | A failslab |
| 118 | |
Vlastimil Babka | 060807f | 2020-08-06 23:18:45 -0700 | [diff] [blame] | 119 | Careful with tracing: It may spew out lots of information and never stop if |
| 120 | used on the wrong slab. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 121 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 122 | Slab merging |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 123 | ============ |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 124 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 125 | If no debug options are specified then SLUB may merge similar slabs together |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 126 | in order to reduce overhead and increase cache hotness of objects. |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 127 | ``slabinfo -a`` displays which slabs were merged together. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 128 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 129 | Slab validation |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 130 | =============== |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 131 | |
| 132 | SLUB can validate all object if the kernel was booted with slub_debug. In |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 133 | order to do so you must have the ``slabinfo`` tool. Then you can do |
| 134 | :: |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 135 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 136 | slabinfo -v |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 137 | |
| 138 | which will test all objects. Output will be generated to the syslog. |
| 139 | |
| 140 | This also works in a more limited way if boot was without slab debug. |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 141 | In that case ``slabinfo -v`` simply tests all reachable objects. Usually |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 142 | these are in the cpu slabs and the partial slabs. Full slabs are not |
| 143 | tracked by SLUB in a non debug situation. |
| 144 | |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 145 | Getting more performance |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 146 | ======================== |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 147 | |
| 148 | To some degree SLUB's performance is limited by the need to take the |
| 149 | list_lock once in a while to deal with partial slabs. That overhead is |
| 150 | governed by the order of the allocation for each slab. The allocations |
| 151 | can be influenced by kernel parameters: |
| 152 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 153 | .. slub_min_objects=x (default 4) |
| 154 | .. slub_min_order=x (default 0) |
| 155 | .. slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER)) |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 156 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 157 | ``slub_min_objects`` |
| 158 | allows to specify how many objects must at least fit into one |
| 159 | slab in order for the allocation order to be acceptable. In |
| 160 | general slub will be able to perform this number of |
| 161 | allocations on a slab without consulting centralized resources |
| 162 | (list_lock) where contention may occur. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 163 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 164 | ``slub_min_order`` |
Tobin C. Harding | 358b6ba | 2019-01-31 15:06:21 +1100 | [diff] [blame] | 165 | specifies a minimum order of slabs. A similar effect like |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 166 | ``slub_min_objects``. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 167 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 168 | ``slub_max_order`` |
| 169 | specified the order at which ``slub_min_objects`` should no |
| 170 | longer be checked. This is useful to avoid SLUB trying to |
| 171 | generate super large order pages to fit ``slub_min_objects`` |
| 172 | of a slab cache with large object sizes into one high order |
| 173 | page. Setting command line parameter |
| 174 | ``debug_guardpage_minorder=N`` (N > 0), forces setting |
| 175 | ``slub_max_order`` to 0, what cause minimum possible order of |
| 176 | slabs allocation. |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 177 | |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 178 | SLUB Debug output |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 179 | ================= |
Christoph Lameter | 3524342 | 2007-05-06 14:49:47 -0700 | [diff] [blame] | 180 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 181 | Here is a sample of slub debug output:: |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 182 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 183 | ==================================================================== |
Kees Cook | 8669dba | 2021-06-15 18:23:19 -0700 | [diff] [blame] | 184 | BUG kmalloc-8: Right Redzone overwritten |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 185 | -------------------------------------------------------------------- |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 186 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 187 | INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc |
| 188 | INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58 |
| 189 | INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58 |
| 190 | INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554 |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 191 | |
Kees Cook | 8669dba | 2021-06-15 18:23:19 -0700 | [diff] [blame] | 192 | Bytes b4 (0xc90f6d10): 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ |
| 193 | Object (0xc90f6d20): 31 30 31 39 2e 30 30 35 1019.005 |
| 194 | Redzone (0xc90f6d28): 00 cc cc cc . |
| 195 | Padding (0xc90f6d50): 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 196 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 197 | [<c010523d>] dump_trace+0x63/0x1eb |
| 198 | [<c01053df>] show_trace_log_lvl+0x1a/0x2f |
| 199 | [<c010601d>] show_trace+0x12/0x14 |
| 200 | [<c0106035>] dump_stack+0x16/0x18 |
| 201 | [<c017e0fa>] object_err+0x143/0x14b |
| 202 | [<c017e2cc>] check_object+0x66/0x234 |
| 203 | [<c017eb43>] __slab_free+0x239/0x384 |
| 204 | [<c017f446>] kfree+0xa6/0xc6 |
| 205 | [<c02e2335>] get_modalias+0xb9/0xf5 |
| 206 | [<c02e23b7>] dmi_dev_uevent+0x27/0x3c |
| 207 | [<c027866a>] dev_uevent+0x1ad/0x1da |
| 208 | [<c0205024>] kobject_uevent_env+0x20a/0x45b |
| 209 | [<c020527f>] kobject_uevent+0xa/0xf |
| 210 | [<c02779f1>] store_uevent+0x4f/0x58 |
| 211 | [<c027758e>] dev_attr_store+0x29/0x2f |
| 212 | [<c01bec4f>] sysfs_write_file+0x16e/0x19c |
| 213 | [<c0183ba7>] vfs_write+0xd1/0x15a |
| 214 | [<c01841d7>] sys_write+0x3d/0x72 |
| 215 | [<c0104112>] sysenter_past_esp+0x5f/0x99 |
| 216 | [<b7f7b410>] 0xb7f7b410 |
| 217 | ======================= |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 218 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 219 | FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 220 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 221 | If SLUB encounters a corrupted object (full detection requires the kernel |
| 222 | to be booted with slub_debug) then the following output will be dumped |
| 223 | into the syslog: |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 224 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 225 | 1. Description of the problem encountered |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 226 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 227 | This will be a message in the system log starting with:: |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 228 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 229 | =============================================== |
| 230 | BUG <slab cache affected>: <What went wrong> |
| 231 | ----------------------------------------------- |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 232 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 233 | INFO: <corruption start>-<corruption_end> <more info> |
| 234 | INFO: Slab <address> <slab information> |
| 235 | INFO: Object <address> <object information> |
| 236 | INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 237 | cpu> pid=<pid of the process> |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 238 | INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu> |
| 239 | pid=<pid of the process> |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 240 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 241 | (Object allocation / free information is only available if SLAB_STORE_USER is |
| 242 | set for the slab. slub_debug sets that option) |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 243 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 244 | 2. The object contents if an object was involved. |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 245 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 246 | Various types of lines can follow the BUG SLUB line: |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 247 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 248 | Bytes b4 <address> : <bytes> |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 249 | Shows a few bytes before the object where the problem was detected. |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 250 | Can be useful if the corruption does not stop with the start of the |
| 251 | object. |
| 252 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 253 | Object <address> : <bytes> |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 254 | The bytes of the object. If the object is inactive then the bytes |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 255 | typically contain poison values. Any non-poison value shows a |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 256 | corruption by a write after free. |
| 257 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 258 | Redzone <address> : <bytes> |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 259 | The Redzone following the object. The Redzone is used to detect |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 260 | writes after the object. All bytes should always have the same |
| 261 | value. If there is any deviation then it is due to a write after |
| 262 | the object boundary. |
| 263 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 264 | (Redzone information is only available if SLAB_RED_ZONE is set. |
| 265 | slub_debug sets that option) |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 266 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 267 | Padding <address> : <bytes> |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 268 | Unused data to fill up the space in order to get the next object |
| 269 | properly aligned. In the debug case we make sure that there are |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 270 | at least 4 bytes of padding. This allows the detection of writes |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 271 | before the object. |
| 272 | |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 273 | 3. A stackdump |
Christoph Lameter | c1aee21 | 2007-05-31 00:40:47 -0700 | [diff] [blame] | 274 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 275 | The stackdump describes the location where the error was detected. The cause |
| 276 | of the corruption is may be more likely found by looking at the function that |
| 277 | allocated or freed the object. |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 278 | |
| 279 | 4. Report on how the problem was dealt with in order to ensure the continued |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 280 | operation of the system. |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 281 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 282 | These are messages in the system log beginning with:: |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 283 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 284 | FIX <slab cache affected>: <corrective action taken> |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 285 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 286 | In the above sample SLUB found that the Redzone of an active object has |
| 287 | been overwritten. Here a string of 8 characters was written into a slab that |
| 288 | has the length of 8 characters. However, a 8 character string needs a |
| 289 | terminating 0. That zero has overwritten the first byte of the Redzone field. |
| 290 | After reporting the details of the issue encountered the FIX SLUB message |
| 291 | tells us that SLUB has restored the Redzone to its proper value and then |
| 292 | system operations continue. |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 293 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 294 | Emergency operations |
| 295 | ==================== |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 296 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 297 | Minimal debugging (sanity checks alone) can be enabled by booting with:: |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 298 | |
| 299 | slub_debug=F |
| 300 | |
| 301 | This will be generally be enough to enable the resiliency features of slub |
| 302 | which will keep the system running even if a bad kernel component will |
| 303 | keep corrupting objects. This may be important for production systems. |
| 304 | Performance will be impacted by the sanity checks and there will be a |
| 305 | continual stream of error messages to the syslog but no additional memory |
| 306 | will be used (unlike full debugging). |
| 307 | |
| 308 | No guarantees. The kernel component still needs to be fixed. Performance |
| 309 | may be optimized further by locating the slab that experiences corruption |
| 310 | and enabling debugging only for that cache |
| 311 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 312 | I.e.:: |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 313 | |
| 314 | slub_debug=F,dentry |
| 315 | |
| 316 | If the corruption occurs by writing after the end of the object then it |
| 317 | may be advisable to enable a Redzone to avoid corrupting the beginning |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 318 | of other objects:: |
Christoph Lameter | 2492268 | 2007-07-17 04:03:18 -0700 | [diff] [blame] | 319 | |
| 320 | slub_debug=FZ,dentry |
| 321 | |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 322 | Extended slabinfo mode and plotting |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 323 | =================================== |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 324 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 325 | The ``slabinfo`` tool has a special 'extended' ('-X') mode that includes: |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 326 | - Slabcache Totals |
| 327 | - Slabs sorted by size (up to -N <num> slabs, default 1) |
| 328 | - Slabs sorted by loss (up to -N <num> slabs, default 1) |
| 329 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 330 | Additionally, in this mode ``slabinfo`` does not dynamically scale |
| 331 | sizes (G/M/K) and reports everything in bytes (this functionality is |
| 332 | also available to other slabinfo modes via '-B' option) which makes |
| 333 | reporting more precise and accurate. Moreover, in some sense the `-X' |
| 334 | mode also simplifies the analysis of slabs' behaviour, because its |
| 335 | output can be plotted using the ``slabinfo-gnuplot.sh`` script. So it |
| 336 | pushes the analysis from looking through the numbers (tons of numbers) |
| 337 | to something easier -- visual analysis. |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 338 | |
| 339 | To generate plots: |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 340 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 341 | a) collect slabinfo extended records, for example:: |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 342 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 343 | while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 344 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 345 | b) pass stats file(-s) to ``slabinfo-gnuplot.sh`` script:: |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 346 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 347 | slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN] |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 348 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 349 | The ``slabinfo-gnuplot.sh`` script will pre-processes the collected records |
| 350 | and generates 3 png files (and 3 pre-processing cache files) per STATS |
| 351 | file: |
| 352 | - Slabcache Totals: FOO_STATS-totals.png |
| 353 | - Slabs sorted by size: FOO_STATS-slabs-by-size.png |
| 354 | - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 355 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 356 | Another use case, when ``slabinfo-gnuplot.sh`` can be useful, is when you |
| 357 | need to compare slabs' behaviour "prior to" and "after" some code |
| 358 | modification. To help you out there, ``slabinfo-gnuplot.sh`` script |
| 359 | can 'merge' the `Slabcache Totals` sections from different |
| 360 | measurements. To visually compare N plots: |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 361 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 362 | a) Collect as many STATS1, STATS2, .. STATSN files as you need:: |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 363 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 364 | while [ 1 ]; do slabinfo -X >> STATS<X>; sleep 1; done |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 365 | |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 366 | b) Pre-process those STATS files:: |
| 367 | |
| 368 | slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN |
| 369 | |
| 370 | c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the |
| 371 | generated pre-processed \*-totals:: |
| 372 | |
| 373 | slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals |
| 374 | |
| 375 | This will produce a single plot (png file). |
| 376 | |
| 377 | Plots, expectedly, can be large so some fluctuations or small spikes |
| 378 | can go unnoticed. To deal with that, ``slabinfo-gnuplot.sh`` has two |
| 379 | options to 'zoom-in'/'zoom-out': |
| 380 | |
Colin Ian King | 94ebdd2 | 2020-10-22 15:26:53 +0100 | [diff] [blame] | 381 | a) ``-s %d,%d`` -- overwrites the default image width and height |
Mike Rapoport | 0c14398 | 2018-03-21 21:22:37 +0200 | [diff] [blame] | 382 | b) ``-r %d,%d`` -- specifies a range of samples to use (for example, |
| 383 | in ``slabinfo -X >> FOO_STATS; sleep 1;`` case, using a ``-r |
| 384 | 40,60`` range will plot only samples collected between 40th and |
| 385 | 60th seconds). |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 386 | |
Christoph Lameter | cde5353 | 2008-07-04 09:59:22 -0700 | [diff] [blame] | 387 | Christoph Lameter, May 30, 2007 |
Sergey Senozhatsky | 05be961 | 2015-10-23 08:51:45 +0900 | [diff] [blame] | 388 | Sergey Senozhatsky, October 23, 2015 |