Functions 3
get_numerical_jacobian_wrt_specific_input
functionFull reference ↗gradcheck
functionFull reference ↗- tensorplay.autograd.gradcheck.gradcheck(func, inputs, *, eps: float = 1e-06, atol: float = 1e-05, rtol: float = 0.001, raise_exception: bool = True, nondet_tol: float = 0.0, check_undefined_grad: bool = True, check_grad_dtypes: bool = False, check_batched_grad: bool = False, check_batched_forward_grad: bool = False, check_forward_ad: bool = False, check_backward_ad: bool = True, fast_mode: bool = False, masked: bool | None = None) bool[source]
Check gradients computed via small finite differences against analytical gradients wrt tensors in
inputsthat are of floating point or complex type and withrequires_grad=True.The check between numerical and analytical gradients uses
allclose().Note
The default values are designed for
inputof double precision. This check will likely fail ifinputis of less precision, e.g.,FloatTensor.Note
Gradcheck may fail when evaluated on non-differentiable points because the numerically computed gradients via finite differencing may differ those computed analytically (not necessarily because either is incorrect).
Warning
If any checked tensor in
inputhas overlapping memory, i.e., different indices pointing to the same memory address (e.g., fromexpand), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address.- Parameters:
func (function) – a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors
eps (float, optional) – perturbation for finite differences
atol (float, optional) – absolute tolerance
rtol (float, optional) – relative tolerance
raise_exception (bool, optional) – indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks.
nondet_tol (float, optional) – tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance.
check_undefined_grad (bool, optional) – if
True, check if undefined output grads are supported and treated as zeros, forTensoroutputs.check_grad_dtypes (bool, optional) – if
True, check that the gradient dtypes match the ones from the numerical computation. Defaults toFalse.check_batched_grad (bool, optional) – Not supported by this engine yet;
TrueraisesNotImplementedError. Defaults to False.check_batched_forward_grad (bool, optional) – Requires forward AD;
TrueraisesNotImplementedError. Defaults to False.check_forward_ad (bool, optional) – Requires forward AD;
TrueraisesNotImplementedError. Defaults to False.check_backward_ad (bool, optional) – if
False, do not perform any checks that rely on backward mode AD to be implemented. Defaults toTrue.fast_mode (bool, optional) – Only the slow implementation exists in this engine;
TrueraisesNotImplementedError. Defaults to False. effect since this engine has no sparse layouts. Defaults toFalse.
- Returns:
Trueif all differences satisfy allclose condition
gradgradcheck
functionFull reference ↗- tensorplay.autograd.gradcheck.gradgradcheck(func, inputs, grad_outputs=None, *, eps: float = 1e-06, atol: float = 1e-05, rtol: float = 0.001, gen_non_contig_grad_outputs: bool = False, raise_exception: bool = True, nondet_tol: float = 0.0, check_undefined_grad: bool = True, check_grad_dtypes: bool = False, check_batched_grad: bool = False, check_fwd_over_rev: bool = False, check_rev_over_rev: bool = True, fast_mode: bool = False, masked: bool = False) bool[source]
Check gradients of gradients computed via small finite differences against analytical gradients wrt tensors in
inputsandgrad_outputsthat are of floating point or complex type and withrequires_grad=True.This function checks that backpropagating through the gradients computed to the given
grad_outputsare correct.The check between numerical and analytical gradients uses
allclose().Note
The default values are designed for
inputandgrad_outputsof double precision. This check will likely fail if they are of less precision, e.g.,FloatTensor.- Parameters:
func (function) – a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors
grad_outputs (tuple of [Tensor or None] or Tensor, optional) – The gradients with respect to the function’s outputs.
eps (float, optional) – perturbation for finite differences
atol (float, optional) – absolute tolerance
rtol (float, optional) – relative tolerance
gen_non_contig_grad_outputs (bool, optional) – Not supported by this engine yet;
TrueraisesNotImplementedError.raise_exception (bool, optional) – indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks.
nondet_tol (float, optional) – tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance. Note that a small amount of nondeterminism in the gradient will lead to larger inaccuracies in the second derivative.
check_undefined_grad (bool, optional) – if True, check if undefined output grads are supported and treated as zeros
check_batched_grad (bool, optional) – Not supported by this engine yet.
fast_mode (bool, optional) – Not supported by this engine yet.
- Returns:
True if all differences satisfy allclose condition
Exceptions 1
GradcheckError
exceptionFull reference ↗- exception tensorplay.autograd.gradcheck.GradcheckError[source]
Error raised by
gradcheck()andgradgradcheck().
