Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 1 | .. SPDX-License-Identifier: GPL-2.0 |
| 2 | |
| 3 | =========== |
| 4 | Using KUnit |
| 5 | =========== |
| 6 | |
| 7 | The purpose of this document is to describe what KUnit is, how it works, how it |
| 8 | is intended to be used, and all the concepts and terminology that are needed to |
| 9 | understand it. This guide assumes a working knowledge of the Linux kernel and |
| 10 | some basic knowledge of testing. |
| 11 | |
| 12 | For a high level introduction to KUnit, including setting up KUnit for your |
| 13 | project, see :doc:`start`. |
| 14 | |
| 15 | Organization of this document |
| 16 | ============================= |
| 17 | |
Daniel Latypov | 1f0e943 | 2020-11-23 14:57:59 -0800 | [diff] [blame] | 18 | This document is organized into two main sections: Testing and Common Patterns. |
| 19 | The first covers what unit tests are and how to use KUnit to write them. The |
| 20 | second covers common testing patterns, e.g. how to isolate code and make it |
| 21 | possible to unit test code that was otherwise un-unit-testable. |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 22 | |
| 23 | Testing |
| 24 | ======= |
| 25 | |
| 26 | What is KUnit? |
| 27 | -------------- |
| 28 | |
| 29 | "K" is short for "kernel" so "KUnit" is the "(Linux) Kernel Unit Testing |
| 30 | Framework." KUnit is intended first and foremost for writing unit tests; it is |
| 31 | general enough that it can be used to write integration tests; however, this is |
| 32 | a secondary goal. KUnit has no ambition of being the only testing framework for |
| 33 | the kernel; for example, it does not intend to be an end-to-end testing |
| 34 | framework. |
| 35 | |
| 36 | What is Unit Testing? |
| 37 | --------------------- |
| 38 | |
| 39 | A `unit test <https://martinfowler.com/bliki/UnitTest.html>`_ is a test that |
| 40 | tests code at the smallest possible scope, a *unit* of code. In the C |
| 41 | programming language that's a function. |
| 42 | |
| 43 | Unit tests should be written for all the publicly exposed functions in a |
| 44 | compilation unit; so that is all the functions that are exported in either a |
| 45 | *class* (defined below) or all functions which are **not** static. |
| 46 | |
| 47 | Writing Tests |
| 48 | ------------- |
| 49 | |
| 50 | Test Cases |
| 51 | ~~~~~~~~~~ |
| 52 | |
| 53 | The fundamental unit in KUnit is the test case. A test case is a function with |
| 54 | the signature ``void (*)(struct kunit *test)``. It calls a function to be tested |
| 55 | and then sets *expectations* for what should happen. For example: |
| 56 | |
| 57 | .. code-block:: c |
| 58 | |
| 59 | void example_test_success(struct kunit *test) |
| 60 | { |
| 61 | } |
| 62 | |
| 63 | void example_test_failure(struct kunit *test) |
| 64 | { |
| 65 | KUNIT_FAIL(test, "This test never passes."); |
| 66 | } |
| 67 | |
| 68 | In the above example ``example_test_success`` always passes because it does |
| 69 | nothing; no expectations are set, so all expectations pass. On the other hand |
| 70 | ``example_test_failure`` always fails because it calls ``KUNIT_FAIL``, which is |
| 71 | a special expectation that logs a message and causes the test case to fail. |
| 72 | |
| 73 | Expectations |
| 74 | ~~~~~~~~~~~~ |
| 75 | An *expectation* is a way to specify that you expect a piece of code to do |
| 76 | something in a test. An expectation is called like a function. A test is made |
| 77 | by setting expectations about the behavior of a piece of code under test; when |
| 78 | one or more of the expectations fail, the test case fails and information about |
| 79 | the failure is logged. For example: |
| 80 | |
| 81 | .. code-block:: c |
| 82 | |
| 83 | void add_test_basic(struct kunit *test) |
| 84 | { |
| 85 | KUNIT_EXPECT_EQ(test, 1, add(1, 0)); |
| 86 | KUNIT_EXPECT_EQ(test, 2, add(1, 1)); |
| 87 | } |
| 88 | |
| 89 | In the above example ``add_test_basic`` makes a number of assertions about the |
| 90 | behavior of a function called ``add``; the first parameter is always of type |
| 91 | ``struct kunit *``, which contains information about the current test context; |
| 92 | the second parameter, in this case, is what the value is expected to be; the |
| 93 | last value is what the value actually is. If ``add`` passes all of these |
| 94 | expectations, the test case, ``add_test_basic`` will pass; if any one of these |
Randy Dunlap | 873ddeb | 2020-10-28 10:43:19 -0700 | [diff] [blame] | 95 | expectations fails, the test case will fail. |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 96 | |
| 97 | It is important to understand that a test case *fails* when any expectation is |
| 98 | violated; however, the test will continue running, potentially trying other |
| 99 | expectations until the test case ends or is otherwise terminated. This is as |
| 100 | opposed to *assertions* which are discussed later. |
| 101 | |
| 102 | To learn about more expectations supported by KUnit, see :doc:`api/test`. |
| 103 | |
| 104 | .. note:: |
| 105 | A single test case should be pretty short, pretty easy to understand, |
| 106 | focused on a single behavior. |
| 107 | |
| 108 | For example, if we wanted to properly test the add function above, we would |
| 109 | create additional tests cases which would each test a different property that an |
| 110 | add function should have like this: |
| 111 | |
| 112 | .. code-block:: c |
| 113 | |
| 114 | void add_test_basic(struct kunit *test) |
| 115 | { |
| 116 | KUNIT_EXPECT_EQ(test, 1, add(1, 0)); |
| 117 | KUNIT_EXPECT_EQ(test, 2, add(1, 1)); |
| 118 | } |
| 119 | |
| 120 | void add_test_negative(struct kunit *test) |
| 121 | { |
| 122 | KUNIT_EXPECT_EQ(test, 0, add(-1, 1)); |
| 123 | } |
| 124 | |
| 125 | void add_test_max(struct kunit *test) |
| 126 | { |
| 127 | KUNIT_EXPECT_EQ(test, INT_MAX, add(0, INT_MAX)); |
| 128 | KUNIT_EXPECT_EQ(test, -1, add(INT_MAX, INT_MIN)); |
| 129 | } |
| 130 | |
| 131 | void add_test_overflow(struct kunit *test) |
| 132 | { |
| 133 | KUNIT_EXPECT_EQ(test, INT_MIN, add(INT_MAX, 1)); |
| 134 | } |
| 135 | |
| 136 | Notice how it is immediately obvious what all the properties that we are testing |
| 137 | for are. |
| 138 | |
| 139 | Assertions |
| 140 | ~~~~~~~~~~ |
| 141 | |
| 142 | KUnit also has the concept of an *assertion*. An assertion is just like an |
| 143 | expectation except the assertion immediately terminates the test case if it is |
| 144 | not satisfied. |
| 145 | |
| 146 | For example: |
| 147 | |
| 148 | .. code-block:: c |
| 149 | |
| 150 | static void mock_test_do_expect_default_return(struct kunit *test) |
| 151 | { |
| 152 | struct mock_test_context *ctx = test->priv; |
| 153 | struct mock *mock = ctx->mock; |
| 154 | int param0 = 5, param1 = -5; |
| 155 | const char *two_param_types[] = {"int", "int"}; |
| 156 | const void *two_params[] = {¶m0, ¶m1}; |
| 157 | const void *ret; |
| 158 | |
| 159 | ret = mock->do_expect(mock, |
| 160 | "test_printk", test_printk, |
| 161 | two_param_types, two_params, |
| 162 | ARRAY_SIZE(two_params)); |
| 163 | KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ret); |
| 164 | KUNIT_EXPECT_EQ(test, -4, *((int *) ret)); |
| 165 | } |
| 166 | |
| 167 | In this example, the method under test should return a pointer to a value, so |
| 168 | if the pointer returned by the method is null or an errno, we don't want to |
| 169 | bother continuing the test since the following expectation could crash the test |
| 170 | case. `ASSERT_NOT_ERR_OR_NULL(...)` allows us to bail out of the test case if |
| 171 | the appropriate conditions have not been satisfied to complete the test. |
| 172 | |
| 173 | Test Suites |
| 174 | ~~~~~~~~~~~ |
| 175 | |
| 176 | Now obviously one unit test isn't very helpful; the power comes from having |
Brendan Higgins | e7d7ad0 | 2019-11-19 15:38:10 -0800 | [diff] [blame] | 177 | many test cases covering all of a unit's behaviors. Consequently it is common |
| 178 | to have many *similar* tests; in order to reduce duplication in these closely |
| 179 | related tests most unit testing frameworks - including KUnit - provide the |
| 180 | concept of a *test suite*. A *test suite* is just a collection of test cases |
| 181 | for a unit of code with a set up function that gets invoked before every test |
| 182 | case and then a tear down function that gets invoked after every test case |
| 183 | completes. |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 184 | |
| 185 | Example: |
| 186 | |
| 187 | .. code-block:: c |
| 188 | |
| 189 | static struct kunit_case example_test_cases[] = { |
| 190 | KUNIT_CASE(example_test_foo), |
| 191 | KUNIT_CASE(example_test_bar), |
| 192 | KUNIT_CASE(example_test_baz), |
| 193 | {} |
| 194 | }; |
| 195 | |
| 196 | static struct kunit_suite example_test_suite = { |
| 197 | .name = "example", |
| 198 | .init = example_test_init, |
| 199 | .exit = example_test_exit, |
| 200 | .test_cases = example_test_cases, |
| 201 | }; |
| 202 | kunit_test_suite(example_test_suite); |
| 203 | |
| 204 | In the above example the test suite, ``example_test_suite``, would run the test |
Randy Dunlap | 873ddeb | 2020-10-28 10:43:19 -0700 | [diff] [blame] | 205 | cases ``example_test_foo``, ``example_test_bar``, and ``example_test_baz``; |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 206 | each would have ``example_test_init`` called immediately before it and would |
| 207 | have ``example_test_exit`` called immediately after it. |
| 208 | ``kunit_test_suite(example_test_suite)`` registers the test suite with the |
| 209 | KUnit test framework. |
| 210 | |
| 211 | .. note:: |
| 212 | A test case will only be run if it is associated with a test suite. |
| 213 | |
Brendan Higgins | a82763e | 2020-08-04 13:47:45 -0700 | [diff] [blame] | 214 | ``kunit_test_suite(...)`` is a macro which tells the linker to put the specified |
| 215 | test suite in a special linker section so that it can be run by KUnit either |
| 216 | after late_init, or when the test module is loaded (depending on whether the |
| 217 | test was built in or not). |
| 218 | |
Brendan Higgins | e7d7ad0 | 2019-11-19 15:38:10 -0800 | [diff] [blame] | 219 | For more information on these types of things see the :doc:`api/test`. |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 220 | |
Daniel Latypov | 1f0e943 | 2020-11-23 14:57:59 -0800 | [diff] [blame] | 221 | Common Patterns |
| 222 | =============== |
| 223 | |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 224 | Isolating Behavior |
Daniel Latypov | 1f0e943 | 2020-11-23 14:57:59 -0800 | [diff] [blame] | 225 | ------------------ |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 226 | |
| 227 | The most important aspect of unit testing that other forms of testing do not |
| 228 | provide is the ability to limit the amount of code under test to a single unit. |
| 229 | In practice, this is only possible by being able to control what code gets run |
| 230 | when the unit under test calls a function and this is usually accomplished |
| 231 | through some sort of indirection where a function is exposed as part of an API |
| 232 | such that the definition of that function can be changed without affecting the |
| 233 | rest of the code base. In the kernel this primarily comes from two constructs, |
| 234 | classes, structs that contain function pointers that are provided by the |
Randy Dunlap | 873ddeb | 2020-10-28 10:43:19 -0700 | [diff] [blame] | 235 | implementer, and architecture-specific functions which have definitions selected |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 236 | at compile time. |
| 237 | |
| 238 | Classes |
Daniel Latypov | 1f0e943 | 2020-11-23 14:57:59 -0800 | [diff] [blame] | 239 | ~~~~~~~ |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 240 | |
| 241 | Classes are not a construct that is built into the C programming language; |
| 242 | however, it is an easily derived concept. Accordingly, pretty much every project |
| 243 | that does not use a standardized object oriented library (like GNOME's GObject) |
| 244 | has their own slightly different way of doing object oriented programming; the |
| 245 | Linux kernel is no exception. |
| 246 | |
| 247 | The central concept in kernel object oriented programming is the class. In the |
| 248 | kernel, a *class* is a struct that contains function pointers. This creates a |
| 249 | contract between *implementers* and *users* since it forces them to use the |
| 250 | same function signature without having to call the function directly. In order |
| 251 | for it to truly be a class, the function pointers must specify that a pointer |
| 252 | to the class, known as a *class handle*, be one of the parameters; this makes |
| 253 | it possible for the member functions (also known as *methods*) to have access |
| 254 | to member variables (more commonly known as *fields*) allowing the same |
| 255 | implementation to have multiple *instances*. |
| 256 | |
| 257 | Typically a class can be *overridden* by *child classes* by embedding the |
| 258 | *parent class* in the child class. Then when a method provided by the child |
| 259 | class is called, the child implementation knows that the pointer passed to it is |
| 260 | of a parent contained within the child; because of this, the child can compute |
| 261 | the pointer to itself because the pointer to the parent is always a fixed offset |
| 262 | from the pointer to the child; this offset is the offset of the parent contained |
| 263 | in the child struct. For example: |
| 264 | |
| 265 | .. code-block:: c |
| 266 | |
| 267 | struct shape { |
| 268 | int (*area)(struct shape *this); |
| 269 | }; |
| 270 | |
| 271 | struct rectangle { |
| 272 | struct shape parent; |
| 273 | int length; |
| 274 | int width; |
| 275 | }; |
| 276 | |
| 277 | int rectangle_area(struct shape *this) |
| 278 | { |
| 279 | struct rectangle *self = container_of(this, struct shape, parent); |
| 280 | |
| 281 | return self->length * self->width; |
| 282 | }; |
| 283 | |
| 284 | void rectangle_new(struct rectangle *self, int length, int width) |
| 285 | { |
| 286 | self->parent.area = rectangle_area; |
| 287 | self->length = length; |
| 288 | self->width = width; |
| 289 | } |
| 290 | |
| 291 | In this example (as in most kernel code) the operation of computing the pointer |
| 292 | to the child from the pointer to the parent is done by ``container_of``. |
| 293 | |
| 294 | Faking Classes |
| 295 | ~~~~~~~~~~~~~~ |
| 296 | |
| 297 | In order to unit test a piece of code that calls a method in a class, the |
| 298 | behavior of the method must be controllable, otherwise the test ceases to be a |
| 299 | unit test and becomes an integration test. |
| 300 | |
| 301 | A fake just provides an implementation of a piece of code that is different than |
| 302 | what runs in a production instance, but behaves identically from the standpoint |
| 303 | of the callers; this is usually done to replace a dependency that is hard to |
| 304 | deal with, or is slow. |
| 305 | |
| 306 | A good example for this might be implementing a fake EEPROM that just stores the |
| 307 | "contents" in an internal buffer. For example, let's assume we have a class that |
| 308 | represents an EEPROM: |
| 309 | |
| 310 | .. code-block:: c |
| 311 | |
| 312 | struct eeprom { |
| 313 | ssize_t (*read)(struct eeprom *this, size_t offset, char *buffer, size_t count); |
| 314 | ssize_t (*write)(struct eeprom *this, size_t offset, const char *buffer, size_t count); |
| 315 | }; |
| 316 | |
| 317 | And we want to test some code that buffers writes to the EEPROM: |
| 318 | |
| 319 | .. code-block:: c |
| 320 | |
| 321 | struct eeprom_buffer { |
| 322 | ssize_t (*write)(struct eeprom_buffer *this, const char *buffer, size_t count); |
| 323 | int flush(struct eeprom_buffer *this); |
| 324 | size_t flush_count; /* Flushes when buffer exceeds flush_count. */ |
| 325 | }; |
| 326 | |
| 327 | struct eeprom_buffer *new_eeprom_buffer(struct eeprom *eeprom); |
| 328 | void destroy_eeprom_buffer(struct eeprom *eeprom); |
| 329 | |
| 330 | We can easily test this code by *faking out* the underlying EEPROM: |
| 331 | |
| 332 | .. code-block:: c |
| 333 | |
| 334 | struct fake_eeprom { |
| 335 | struct eeprom parent; |
| 336 | char contents[FAKE_EEPROM_CONTENTS_SIZE]; |
| 337 | }; |
| 338 | |
| 339 | ssize_t fake_eeprom_read(struct eeprom *parent, size_t offset, char *buffer, size_t count) |
| 340 | { |
| 341 | struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent); |
| 342 | |
| 343 | count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset); |
| 344 | memcpy(buffer, this->contents + offset, count); |
| 345 | |
| 346 | return count; |
| 347 | } |
| 348 | |
Brendan Higgins | e7d7ad0 | 2019-11-19 15:38:10 -0800 | [diff] [blame] | 349 | ssize_t fake_eeprom_write(struct eeprom *parent, size_t offset, const char *buffer, size_t count) |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 350 | { |
| 351 | struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent); |
| 352 | |
| 353 | count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset); |
| 354 | memcpy(this->contents + offset, buffer, count); |
| 355 | |
| 356 | return count; |
| 357 | } |
| 358 | |
| 359 | void fake_eeprom_init(struct fake_eeprom *this) |
| 360 | { |
| 361 | this->parent.read = fake_eeprom_read; |
| 362 | this->parent.write = fake_eeprom_write; |
| 363 | memset(this->contents, 0, FAKE_EEPROM_CONTENTS_SIZE); |
| 364 | } |
| 365 | |
| 366 | We can now use it to test ``struct eeprom_buffer``: |
| 367 | |
| 368 | .. code-block:: c |
| 369 | |
| 370 | struct eeprom_buffer_test { |
| 371 | struct fake_eeprom *fake_eeprom; |
| 372 | struct eeprom_buffer *eeprom_buffer; |
| 373 | }; |
| 374 | |
| 375 | static void eeprom_buffer_test_does_not_write_until_flush(struct kunit *test) |
| 376 | { |
| 377 | struct eeprom_buffer_test *ctx = test->priv; |
| 378 | struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; |
| 379 | struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; |
| 380 | char buffer[] = {0xff}; |
| 381 | |
| 382 | eeprom_buffer->flush_count = SIZE_MAX; |
| 383 | |
| 384 | eeprom_buffer->write(eeprom_buffer, buffer, 1); |
| 385 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); |
| 386 | |
| 387 | eeprom_buffer->write(eeprom_buffer, buffer, 1); |
| 388 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0); |
| 389 | |
| 390 | eeprom_buffer->flush(eeprom_buffer); |
| 391 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); |
| 392 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); |
| 393 | } |
| 394 | |
| 395 | static void eeprom_buffer_test_flushes_after_flush_count_met(struct kunit *test) |
| 396 | { |
| 397 | struct eeprom_buffer_test *ctx = test->priv; |
| 398 | struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; |
| 399 | struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; |
| 400 | char buffer[] = {0xff}; |
| 401 | |
| 402 | eeprom_buffer->flush_count = 2; |
| 403 | |
| 404 | eeprom_buffer->write(eeprom_buffer, buffer, 1); |
| 405 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); |
| 406 | |
| 407 | eeprom_buffer->write(eeprom_buffer, buffer, 1); |
| 408 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); |
| 409 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); |
| 410 | } |
| 411 | |
| 412 | static void eeprom_buffer_test_flushes_increments_of_flush_count(struct kunit *test) |
| 413 | { |
| 414 | struct eeprom_buffer_test *ctx = test->priv; |
| 415 | struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; |
| 416 | struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; |
| 417 | char buffer[] = {0xff, 0xff}; |
| 418 | |
| 419 | eeprom_buffer->flush_count = 2; |
| 420 | |
| 421 | eeprom_buffer->write(eeprom_buffer, buffer, 1); |
| 422 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); |
| 423 | |
| 424 | eeprom_buffer->write(eeprom_buffer, buffer, 2); |
| 425 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); |
| 426 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); |
| 427 | /* Should have only flushed the first two bytes. */ |
| 428 | KUNIT_EXPECT_EQ(test, fake_eeprom->contents[2], 0); |
| 429 | } |
| 430 | |
| 431 | static int eeprom_buffer_test_init(struct kunit *test) |
| 432 | { |
| 433 | struct eeprom_buffer_test *ctx; |
| 434 | |
| 435 | ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); |
| 436 | KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx); |
| 437 | |
| 438 | ctx->fake_eeprom = kunit_kzalloc(test, sizeof(*ctx->fake_eeprom), GFP_KERNEL); |
| 439 | KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->fake_eeprom); |
| 440 | fake_eeprom_init(ctx->fake_eeprom); |
| 441 | |
| 442 | ctx->eeprom_buffer = new_eeprom_buffer(&ctx->fake_eeprom->parent); |
| 443 | KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->eeprom_buffer); |
| 444 | |
| 445 | test->priv = ctx; |
| 446 | |
| 447 | return 0; |
| 448 | } |
| 449 | |
| 450 | static void eeprom_buffer_test_exit(struct kunit *test) |
| 451 | { |
| 452 | struct eeprom_buffer_test *ctx = test->priv; |
| 453 | |
| 454 | destroy_eeprom_buffer(ctx->eeprom_buffer); |
| 455 | } |
| 456 | |
Daniel Latypov | 1f0e943 | 2020-11-23 14:57:59 -0800 | [diff] [blame] | 457 | Testing against multiple inputs |
| 458 | ------------------------------- |
| 459 | |
| 460 | Testing just a few inputs might not be enough to have confidence that the code |
| 461 | works correctly, e.g. for a hash function. |
| 462 | |
| 463 | In such cases, it can be helpful to have a helper macro or function, e.g. this |
| 464 | fictitious example for ``sha1sum(1)`` |
| 465 | |
| 466 | .. code-block:: c |
| 467 | |
| 468 | /* Note: the cast is to satisfy overly strict type-checking. */ |
| 469 | #define TEST_SHA1(in, want) \ |
| 470 | sha1sum(in, out); \ |
| 471 | KUNIT_EXPECT_STREQ_MSG(test, (char *)out, want, "sha1sum(%s)", in); |
| 472 | |
| 473 | char out[40]; |
| 474 | TEST_SHA1("hello world", "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed"); |
| 475 | TEST_SHA1("hello world!", "430ce34d020724ed75a196dfc2ad67c77772d169"); |
| 476 | |
| 477 | |
| 478 | Note the use of ``KUNIT_EXPECT_STREQ_MSG`` to give more context when it fails |
| 479 | and make it easier to track down. (Yes, in this example, ``want`` is likely |
| 480 | going to be unique enough on its own). |
| 481 | |
| 482 | The ``_MSG`` variants are even more useful when the same expectation is called |
| 483 | multiple times (in a loop or helper function) and thus the line number isn't |
| 484 | enough to identify what failed, like below. |
| 485 | |
| 486 | In some cases, it can be helpful to write a *table-driven test* instead, e.g. |
| 487 | |
| 488 | .. code-block:: c |
| 489 | |
| 490 | int i; |
| 491 | char out[40]; |
| 492 | |
| 493 | struct sha1_test_case { |
| 494 | const char *str; |
| 495 | const char *sha1; |
| 496 | }; |
| 497 | |
| 498 | struct sha1_test_case cases[] = { |
| 499 | { |
| 500 | .str = "hello world", |
| 501 | .sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", |
| 502 | }, |
| 503 | { |
| 504 | .str = "hello world!", |
| 505 | .sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169", |
| 506 | }, |
| 507 | }; |
| 508 | for (i = 0; i < ARRAY_SIZE(cases); ++i) { |
| 509 | sha1sum(cases[i].str, out); |
| 510 | KUNIT_EXPECT_STREQ_MSG(test, (char *)out, cases[i].sha1, |
| 511 | "sha1sum(%s)", cases[i].str); |
| 512 | } |
| 513 | |
| 514 | |
| 515 | There's more boilerplate involved, but it can: |
| 516 | |
| 517 | * be more readable when there are multiple inputs/outputs thanks to field names, |
| 518 | |
| 519 | * E.g. see ``fs/ext4/inode-test.c`` for an example of both. |
| 520 | * reduce duplication if test cases can be shared across multiple tests. |
| 521 | |
| 522 | * E.g. if we wanted to also test ``sha256sum``, we could add a ``sha256`` |
| 523 | field and reuse ``cases``. |
| 524 | |
Daniel Latypov | 8db50be | 2020-12-15 16:22:46 -0800 | [diff] [blame] | 525 | * be converted to a "parameterized test", see below. |
| 526 | |
| 527 | Parameterized Testing |
| 528 | ~~~~~~~~~~~~~~~~~~~~~ |
| 529 | |
| 530 | The table-driven testing pattern is common enough that KUnit has special |
| 531 | support for it. |
| 532 | |
| 533 | Reusing the same ``cases`` array from above, we can write the test as a |
| 534 | "parameterized test" with the following. |
| 535 | |
| 536 | .. code-block:: c |
| 537 | |
| 538 | // This is copy-pasted from above. |
| 539 | struct sha1_test_case { |
| 540 | const char *str; |
| 541 | const char *sha1; |
| 542 | }; |
| 543 | struct sha1_test_case cases[] = { |
| 544 | { |
| 545 | .str = "hello world", |
| 546 | .sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", |
| 547 | }, |
| 548 | { |
| 549 | .str = "hello world!", |
| 550 | .sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169", |
| 551 | }, |
| 552 | }; |
| 553 | |
| 554 | // Need a helper function to generate a name for each test case. |
| 555 | static void case_to_desc(const struct sha1_test_case *t, char *desc) |
| 556 | { |
| 557 | strcpy(desc, t->str); |
| 558 | } |
| 559 | // Creates `sha1_gen_params()` to iterate over `cases`. |
| 560 | KUNIT_ARRAY_PARAM(sha1, cases, case_to_desc); |
| 561 | |
| 562 | // Looks no different from a normal test. |
| 563 | static void sha1_test(struct kunit *test) |
| 564 | { |
| 565 | // This function can just contain the body of the for-loop. |
| 566 | // The former `cases[i]` is accessible under test->param_value. |
| 567 | char out[40]; |
| 568 | struct sha1_test_case *test_param = (struct sha1_test_case *)(test->param_value); |
| 569 | |
| 570 | sha1sum(test_param->str, out); |
| 571 | KUNIT_EXPECT_STREQ_MSG(test, (char *)out, test_param->sha1, |
| 572 | "sha1sum(%s)", test_param->str); |
| 573 | } |
| 574 | |
| 575 | // Instead of KUNIT_CASE, we use KUNIT_CASE_PARAM and pass in the |
| 576 | // function declared by KUNIT_ARRAY_PARAM. |
| 577 | static struct kunit_case sha1_test_cases[] = { |
| 578 | KUNIT_CASE_PARAM(sha1_test, sha1_gen_params), |
| 579 | {} |
| 580 | }; |
| 581 | |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 582 | .. _kunit-on-non-uml: |
| 583 | |
| 584 | KUnit on non-UML architectures |
| 585 | ============================== |
| 586 | |
| 587 | By default KUnit uses UML as a way to provide dependencies for code under test. |
| 588 | Under most circumstances KUnit's usage of UML should be treated as an |
| 589 | implementation detail of how KUnit works under the hood. Nevertheless, there |
Randy Dunlap | 873ddeb | 2020-10-28 10:43:19 -0700 | [diff] [blame] | 590 | are instances where being able to run architecture-specific code or test |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 591 | against real hardware is desirable. For these reasons KUnit supports running on |
| 592 | other architectures. |
| 593 | |
| 594 | Running existing KUnit tests on non-UML architectures |
| 595 | ----------------------------------------------------- |
| 596 | |
| 597 | There are some special considerations when running existing KUnit tests on |
| 598 | non-UML architectures: |
| 599 | |
| 600 | * Hardware may not be deterministic, so a test that always passes or fails |
| 601 | when run under UML may not always do so on real hardware. |
| 602 | * Hardware and VM environments may not be hermetic. KUnit tries its best to |
| 603 | provide a hermetic environment to run tests; however, it cannot manage state |
| 604 | that it doesn't know about outside of the kernel. Consequently, tests that |
| 605 | may be hermetic on UML may not be hermetic on other architectures. |
| 606 | * Some features and tooling may not be supported outside of UML. |
| 607 | * Hardware and VMs are slower than UML. |
| 608 | |
| 609 | None of these are reasons not to run your KUnit tests on real hardware; they are |
| 610 | only things to be aware of when doing so. |
| 611 | |
Brendan Higgins | 12ca7a8 | 2021-05-26 14:24:07 -0700 | [diff] [blame^] | 612 | Currently, the KUnit Wrapper (``tools/testing/kunit/kunit.py``) (aka |
| 613 | kunit_tool) only fully supports running tests inside of UML and QEMU; however, |
| 614 | this is only due to our own time limitations as humans working on KUnit. It is |
| 615 | entirely possible to support other emulators and even actual hardware, but for |
| 616 | now QEMU and UML is what is fully supported within the KUnit Wrapper. Again, to |
| 617 | be clear, this is just the Wrapper. The actualy KUnit tests and the KUnit |
| 618 | library they are written in is fully architecture agnostic and can be used in |
| 619 | virtually any setup, you just won't have the benefit of typing a single command |
| 620 | out of the box and having everything magically work perfectly. |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 621 | |
Brendan Higgins | 12ca7a8 | 2021-05-26 14:24:07 -0700 | [diff] [blame^] | 622 | Again, all core KUnit framework features are fully supported on all |
| 623 | architectures, and using them is straightforward: Most popular architectures |
| 624 | are supported directly in the KUnit Wrapper via QEMU. Currently, supported |
| 625 | architectures on QEMU include: |
| 626 | |
| 627 | * i386 |
| 628 | * x86_64 |
| 629 | * arm |
| 630 | * arm64 |
| 631 | * alpha |
| 632 | * powerpc |
| 633 | * riscv |
| 634 | * s390 |
| 635 | * sparc |
| 636 | |
| 637 | In order to run KUnit tests on one of these architectures via QEMU with the |
| 638 | KUnit wrapper, all you need to do is specify the flags ``--arch`` and |
| 639 | ``--cross_compile`` when invoking the KUnit Wrapper. For example, we could run |
| 640 | the default KUnit tests on ARM in the following manner (assuming we have an ARM |
| 641 | toolchain installed): |
| 642 | |
| 643 | .. code-block:: bash |
| 644 | |
| 645 | tools/testing/kunit/kunit.py run --timeout=60 --jobs=12 --arch=arm --cross_compile=arm-linux-gnueabihf- |
| 646 | |
| 647 | Alternatively, if you want to run your tests on real hardware or in some other |
| 648 | emulation environment, all you need to do is to take your kunitconfig, your |
| 649 | Kconfig options for the tests you would like to run, and merge them into |
| 650 | whatever config your are using for your platform. That's it! |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 651 | |
| 652 | For example, let's say you have the following kunitconfig: |
| 653 | |
| 654 | .. code-block:: none |
| 655 | |
| 656 | CONFIG_KUNIT=y |
| 657 | CONFIG_KUNIT_EXAMPLE_TEST=y |
| 658 | |
| 659 | If you wanted to run this test on an x86 VM, you might add the following config |
| 660 | options to your ``.config``: |
| 661 | |
| 662 | .. code-block:: none |
| 663 | |
| 664 | CONFIG_KUNIT=y |
| 665 | CONFIG_KUNIT_EXAMPLE_TEST=y |
| 666 | CONFIG_SERIAL_8250=y |
| 667 | CONFIG_SERIAL_8250_CONSOLE=y |
| 668 | |
| 669 | All these new options do is enable support for a common serial console needed |
| 670 | for logging. |
| 671 | |
| 672 | Next, you could build a kernel with these tests as follows: |
| 673 | |
| 674 | |
| 675 | .. code-block:: bash |
| 676 | |
| 677 | make ARCH=x86 olddefconfig |
| 678 | make ARCH=x86 |
| 679 | |
| 680 | Once you have built a kernel, you could run it on QEMU as follows: |
| 681 | |
| 682 | .. code-block:: bash |
| 683 | |
| 684 | qemu-system-x86_64 -enable-kvm \ |
| 685 | -m 1024 \ |
| 686 | -kernel arch/x86_64/boot/bzImage \ |
| 687 | -append 'console=ttyS0' \ |
| 688 | --nographic |
| 689 | |
| 690 | Interspersed in the kernel logs you might see the following: |
| 691 | |
| 692 | .. code-block:: none |
| 693 | |
| 694 | TAP version 14 |
| 695 | # Subtest: example |
| 696 | 1..1 |
| 697 | # example_simple_test: initializing |
| 698 | ok 1 - example_simple_test |
| 699 | ok 1 - example |
| 700 | |
| 701 | Congratulations, you just ran a KUnit test on the x86 architecture! |
| 702 | |
Alan Maguire | 6ae2bfd | 2020-01-06 22:28:23 +0000 | [diff] [blame] | 703 | In a similar manner, kunit and kunit tests can also be built as modules, |
| 704 | so if you wanted to run tests in this way you might add the following config |
| 705 | options to your ``.config``: |
| 706 | |
| 707 | .. code-block:: none |
| 708 | |
| 709 | CONFIG_KUNIT=m |
| 710 | CONFIG_KUNIT_EXAMPLE_TEST=m |
| 711 | |
| 712 | Once the kernel is built and installed, a simple |
| 713 | |
| 714 | .. code-block:: bash |
Brendan Higgins | e20d8e8 | 2020-01-31 16:01:02 -0800 | [diff] [blame] | 715 | |
Alan Maguire | 6ae2bfd | 2020-01-06 22:28:23 +0000 | [diff] [blame] | 716 | modprobe example-test |
| 717 | |
| 718 | ...will run the tests. |
| 719 | |
SeongJae Park | f0b6203 | 2020-10-21 21:25:18 +0200 | [diff] [blame] | 720 | .. note:: |
| 721 | Note that you should make sure your test depends on ``KUNIT=y`` in Kconfig |
| 722 | if the test does not support module build. Otherwise, it will trigger |
| 723 | compile errors if ``CONFIG_KUNIT`` is ``m``. |
| 724 | |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 725 | Writing new tests for other architectures |
| 726 | ----------------------------------------- |
| 727 | |
| 728 | The first thing you must do is ask yourself whether it is necessary to write a |
| 729 | KUnit test for a specific architecture, and then whether it is necessary to |
| 730 | write that test for a particular piece of hardware. In general, writing a test |
| 731 | that depends on having access to a particular piece of hardware or software (not |
| 732 | included in the Linux source repo) should be avoided at all costs. |
| 733 | |
| 734 | Even if you only ever plan on running your KUnit test on your hardware |
| 735 | configuration, other people may want to run your tests and may not have access |
| 736 | to your hardware. If you write your test to run on UML, then anyone can run your |
| 737 | tests without knowing anything about your particular setup, and you can still |
| 738 | run your tests on your hardware setup just by compiling for your architecture. |
| 739 | |
| 740 | .. important:: |
| 741 | Always prefer tests that run on UML to tests that only run under a particular |
| 742 | architecture, and always prefer tests that run under QEMU or another easy |
Brendan Higgins | e7d7ad0 | 2019-11-19 15:38:10 -0800 | [diff] [blame] | 743 | (and monetarily free) to obtain software environment to a specific piece of |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 744 | hardware. |
| 745 | |
| 746 | Nevertheless, there are still valid reasons to write an architecture or hardware |
| 747 | specific test: for example, you might want to test some code that really belongs |
| 748 | in ``arch/some-arch/*``. Even so, try your best to write the test so that it |
| 749 | does not depend on physical hardware: if some of your test cases don't need the |
| 750 | hardware, only require the hardware for tests that actually need it. |
| 751 | |
| 752 | Now that you have narrowed down exactly what bits are hardware specific, the |
| 753 | actual procedure for writing and running the tests is pretty much the same as |
| 754 | writing normal KUnit tests. One special caveat is that you have to reset |
| 755 | hardware state in between test cases; if this is not possible, you may only be |
| 756 | able to run one test case per invocation. |
| 757 | |
Randy Dunlap | 873ddeb | 2020-10-28 10:43:19 -0700 | [diff] [blame] | 758 | .. TODO(brendanhiggins@google.com): Add an actual example of an architecture- |
Brendan Higgins | c23a283 | 2019-09-23 02:02:45 -0700 | [diff] [blame] | 759 | dependent KUnit test. |
Alan Maguire | 3252690 | 2020-03-26 14:25:10 +0000 | [diff] [blame] | 760 | |
| 761 | KUnit debugfs representation |
| 762 | ============================ |
| 763 | When kunit test suites are initialized, they create an associated directory |
Lothar Rubusch | c4714b0 | 2020-04-15 20:16:53 +0000 | [diff] [blame] | 764 | in ``/sys/kernel/debug/kunit/<test-suite>``. The directory contains one file |
Alan Maguire | 3252690 | 2020-03-26 14:25:10 +0000 | [diff] [blame] | 765 | |
| 766 | - results: "cat results" displays results of each test case and the results |
| 767 | of the entire suite for the last test run. |
| 768 | |
| 769 | The debugfs representation is primarily of use when kunit test suites are |
| 770 | run in a native environment, either as modules or builtin. Having a way |
| 771 | to display results like this is valuable as otherwise results can be |
| 772 | intermixed with other events in dmesg output. The maximum size of each |
Lothar Rubusch | c4714b0 | 2020-04-15 20:16:53 +0000 | [diff] [blame] | 773 | results file is KUNIT_LOG_SIZE bytes (defined in ``include/kunit/test.h``). |