syms a
syms b
B=[-sin(a) -sin(b)*cos(a) cos(b)*cos(a);cos(a) -sin(b)*sin(a),cos(b)*sin(a);0 cos(b) sin(b)];
inv(B)
inv(B)*B;

检查是否是正确的，只需把syms代入具体的值：

a=pi/2
b=pi/3
B=[-sin(a) -sin(b)*cos(a) cos(b)*cos(a);cos(a) -sin(b)*sin(a),cos(b)*sin(a);0 cos(b) sin(b)];
inv(B)
inv(B)*B