What you need is a quadrangle instead of a rotated rectangle. RotatedRect will give you incorrect results. Also you will need a perspective projection.

Basicly what must been done is:

• Loop through all polygon segments and connect those which are almost equel.
• Sort them so you have the 4 most largest line segments.
• Intersect those lines and you have the 4 most likely corner points.
• Transform the matrix over the perspective gathered from the corner points and the aspect ratio of the known object.

I implemented a class Quadrangle which takes care of contour to quadrangle conversion and will also transform it over the right perspective.

See a working implementation here: Java OpenCV deskewing a contour

What you need is a quadrangle instead of a rotated rectangle. RotatedRect will give you incorrect results. Also you will need a perspective projection.

Basicly what must been done is:

• Loop through all polygon segments and connect those which are almost equel.
• Sort them so you have the 4 most largest line segments.
• Intersect those lines and you have the 4 most likely corner points.
• Transform the matrix over the perspective gathered from the corner points and the aspect ratio of the known object.

I implemented a class Quadrangle which takes care of contour to quadrangle conversion and will also transform it over the right perspective.

See a working implementation here: Java OpenCV deskewing a contour