tensorplay.autograd.gradcheck.gradcheck

tensorplay.autograd.gradcheck.gradcheck(func, inputs, *, eps: float = 1e-06, atol: float = 1e-05, rtol: float = 0.001, raise_exception: bool = True, nondet_tol: float = 0.0, check_undefined_grad: bool = True, check_grad_dtypes: bool = False, check_batched_grad: bool = False, check_batched_forward_grad: bool = False, check_forward_ad: bool = False, check_backward_ad: bool = True, fast_mode: bool = False, masked: bool | None = None) → bool[source]

Check gradients computed via small finite differences against analytical gradients wrt tensors in inputs that are of floating point or complex type and with requires_grad=True.

The check between numerical and analytical gradients uses allclose().

Note

The default values are designed for input of double precision. This check will likely fail if input is of less precision, e.g., FloatTensor.

Note

Gradcheck may fail when evaluated on non-differentiable points because the numerically computed gradients via finite differencing may differ those computed analytically (not necessarily because either is incorrect).

Warning

If any checked tensor in input has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from expand), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address.

Parameters:

• func (function) – a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors

• inputs (tuple of Tensor or Tensor) – inputs to the function

• eps (float, optional) – perturbation for finite differences

• atol (float, optional) – absolute tolerance

• rtol (float, optional) – relative tolerance

• raise_exception (bool, optional) – indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks.

• nondet_tol (float, optional) – tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance.

• check_undefined_grad (bool, optional) – if True, check if undefined output grads are supported and treated as zeros, for Tensor outputs.

• check_grad_dtypes (bool, optional) – if True, check that the gradient dtypes match the ones from the numerical computation. Defaults to False.

• check_batched_grad (bool, optional) – Not supported by this engine yet; True raises NotImplementedError. Defaults to False.

• check_batched_forward_grad (bool, optional) – Requires forward AD; True raises NotImplementedError. Defaults to False.

• check_forward_ad (bool, optional) – Requires forward AD; True raises NotImplementedError. Defaults to False.

• check_backward_ad (bool, optional) – if False, do not perform any checks that rely on backward mode AD to be implemented. Defaults to True.

• fast_mode (bool, optional) – Only the slow implementation exists in this engine; True raises NotImplementedError. Defaults to False. effect since this engine has no sparse layouts. Defaults to False.

Returns:

True if all differences satisfy allclose condition