This field's ownership is unclear: it's in wlr_input_device, but
it's not managed by the common code, it's up to each individual
backend to use it and clean it up.
Since this is a backend implementation detail, move it to the
backend-specific structs.