How can I detect rotated images/documents?

[denivic]

Hello,

I'm sort of new to AI. While I have had two courses about machine learning at university, I never really applied the knowledge outside of the class and since it's been 2 years since I finished my BSc, I kind of forgot most of it. I hope you guys can help me.

I would like to detect whether a document has been rotated or not, which I believe is what db_resnet50_rotation is for, right? Well, I have had some success using this model for extracting text on a rotated image, but the extracted strings often don't match the text on the document. I'm unsure of how I can improve the accuracy, so what I would like to do instead is detect whether a PDF is rotated and if it is then I would like to change its rotation such that it's "straight" after which I will use EasyOCR, which I've had great success with.

I have tried the following using PDFs that I converted to images:

def get_orientation(self, image):
predictor = detection_predictor('db_resnet50_rotation',
                                pretrained=True,
                                assume_straight_pages=False,
                                preserve_aspect_ratio=True)
result = predictor(image)
return result

I was expecting to get a normalized value between 0 and 1, but instead I get a multidimensional array with float values. Initially I thought it returned the image itself so I tried to display it, but on second thought it doesn't make sense to return an image. So what am I missing here? What am I doing wrong?

I tried to look at the documentation, but the documentation wasn't all that helpful. It says detection_predictorreturns a predictor. But how is predictor used and what can I expect it to return? It doesn't say that.

My current solution is to use PyTesseract, but I would prefer to use one module rather than multiple:

def get_orientation(self, path: str) -> int:
    metadata = pytesseract.image_to_osd(path)
    orientation = search('(?<=Rotate: )\d+', metadata).group(0)
    return orientation

Ideally I would like to train my own model on the documents we use as they have a specific layout and training on them might improve our results. We have a server available with AI-centric GPUs. This might be more appropriate in another thread though.

Side note: I don't know about you, but the name makes it hard to search for help as Google constantly correct it to "doctor". Even with quotation marks it is still hard to find answers.

[felixdittrich92]

Hi @denivic 👋,

normally from my experience docTR works in the most cases more accure as EasyOCR would it be possible to share an example image to see what's going wrong ? :)

If you want to get the document orientation you can simply pass detect_orientation=True to the ocr_predictor (with ocr) otherwise you can use the following function:

doctr/doctr/models/_utils.py

Line 29 in 3deac68

def estimate_orientation(img: np.ndarray, n_ct: int = 50, ratio_threshold_for_lines: float = 5) -> float:

The detection_predictor returns the normalized boxes -> (113, 4, 2) (num_boxes, (coords - [[x1 y1], [x2 y2], [x3 y3], [x4 y4]])) to restore the values you need to multiply the values with the corresponding page dimensions (width, height).

If you can train your own models see: https://mindee.github.io/doctr/latest/using_doctr/custom_models_training.html

Best regards,
Felix

[denivic]

Hey @felixdittrich92!

Thank you for the informative reply, I appreciate it. As for DocTR being more efficient than EasyOCR that might be true, I forgot to mention that the approach I'm using is:

1. crop sections of PDF into small images.
2. Read with OCR.

This probably explains why I didn't have issues with EasyOCR. I forgot to crop the image when using DocTR 🤦‍♂️. I'll give it another shot with DocTR (especially now that you mentioned detect_orientation=True - thanks for that) but with cropped images this time.

I would love to share an example of the files that I'm working with, but I am unable to as they contain sensitive information about other people such as their name, SSN, and the university they study at. I work for a specific ministry/branch of my government that handles education and research in my country. These documents are sent in by students who study abroad, but still want the monthly government "allowance" so to speak. Students are paid to study here ranging from ~150€ to ~740€ depending on their situation (whether they live at home or on their own, how much their parents earn etc). I could perhaps censor the sensitive info or simply overwrite it with fake data. I think I'll try your suggested solutions first and if it still doesn't work I'll prepare some examples along with the code I ended up writing. :)

You can see an example of an empty form here (found on su.dk at the bottom of the page section "Studieaktivitet - krav til dokumentation").

[felixT2K]

Hi @denivic i am on it see #1336
I will update you soon :)

[felixdittrich92]

@denivic It's still in progress but already testable so you can checkout to the branch and test it i'm happy about every feedback :)

[felixdittrich92]

Btw. for the 0.7.1 release we don't have a fixed deadline but the goal is to do the release end of november/ start december

[denivic]

Hi @denivic i am on it see #1336 I will update you soon :)

Amazing, thank you so much! 😁

Btw. for the 0.7.1 release we don't have a fixed deadline but the goal is to do the release end of november/ start december

Cool, I'm looking forward to it.

[felixdittrich92]

@denivic Could you test it on the branch ?

It should work much better for rotated samples in a range of (-90 - 90 degrees)

• will further improve with the new models (because now depends on the detection results)
• upside down detection is still a todo task

[denivic]

@felixdittrich92 I have tested #1336 on a sample size of 21 documents with random angles between -90 and 90 and it seems to not be working as expected. I might be doing something wrong, but this is what I'm getting:

1. Orientation found: 14.845458984375, Actual orientation: -3
2. Orientation found: 10.808235168457031, Actual orientation: 3
3. Orientation found: -73.0724868774414, Actual orientation: 19
4. Orientation found: 39.805572509765625, Actual orientation: -70
5. Orientation found: 86.9471173286438, Actual orientation: 48
6. Orientation found: 12.90740966796875, Actual orientation: 71
7. Orientation found: -42.41420364379883, Actual orientation: -32
8. Orientation found: 0.6437454223632812, Actual orientation: -28
9. Orientation found: -6.009005546569824, Actual orientation: 15
10. Orientation found: -49.39870071411133, Actual orientation: -4
11. Orientation found: -52.177249908447266, Actual orientation: 4
12. Orientation found: 48.90570068359375, Actual orientation: 6
13. Orientation found: -45.364933013916016, Actual orientation: 74
14. Orientation found: 26.113914489746094, Actual orientation: -78
15. Orientation found: 45.88140106201172, Actual orientation: 85
16. Orientation found: 79.5085220336914, Actual orientation: 83
17. Orientation found: 60.80251693725586, Actual orientation: -7
18. Orientation found: 0, Actual orientation: -40
19. Orientation found: -8.310276985168457, Actual orientation: 62
20. Orientation found: -14.444034576416016, Actual orientation: 90
21. Orientation found: -9.090276718139648, Actual orientation: -43

Using these methods. It could possibly be that I'm iterating through my list of randomly selected angles the wrong way with respect to the list of images, but I don't think that I am though unless ocr_predictor returns in an unordered fashion then these results might be wrong. I can see that some of these match (or are at the very least very close) such as the actual rotation of number 5 matching the orientation found for number 12. I might be reading too much into this though, it could very well be a coincidence.

EDIT: Well... Seems like some of the images are rotated more than 90 or -90 degrees for some reason causing them to be almost upside down, which is weird because I checked the images manually before loading them using DocumentFile.from_images(). Some of these results are still accurate (accurate in the sense that the images are actually within the -90 and 90 range yet the wrong orientation is reported). I'll re-do these tests tomorrow and verify manually that each and every image that is being tested is within -90 and 90.

[denivic]

I get that it's an OCR predictor, but I suppose I assumed (incorrectly I guess) that it'd still compute the orientation despite no text being present and just output None in regards to text. Is it computing the orientation using the text itself?

Here is the PDF that doesn't work when I convert it to a .png image: https://www.su.dk/media/1239/dansk.pdf

[felixdittrich92]

Correct we compute the orientation on the median line of the segmentation map which we get from the detection module

[felixdittrich92]

I have tested it with your provided pdf and duplicated it 20 times to apply random rotation.

Code PyTorch:

import random
import time

import requests

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
from doctr.utils.geometry import rotate_image

#bytes_data = requests.get(
#    "https://www.allianzdirect.de/dam/documents/home/Versicherungsbedingungen-08-2021.pdf"
#).content

start = time.time()
# doc = DocumentFile.from_pdf("/home/felix/Desktop/vitstr_paper.pdf")
#doc = DocumentFile.from_pdf(bytes_data)
# doc = DocumentFile.from_images(['/home/felix/Desktop/5.png']) # * 20)
doc = DocumentFile.from_pdf('/home/felix/Desktop/dansk.pdf') * 20
rnds = [random.randint(-90, 90) for _ in range(len(doc))]
# rnds = [random.choice([90, -90]) for _ in range(len(doc))]
doc = [rotate_image(page, rnd, expand=True) for page, rnd in zip(doc, rnds)]

model = ocr_predictor(
    det_arch="db_resnet50_rotation",
    reco_arch="parseq",
    pretrained=True,
    det_bs=8,
    reco_bs=1024,
    assume_straight_pages=False,
    straighten_pages=True,
    detect_orientation=True,
).cuda().half()  # uncomment if you have a GPU

result = model(doc)
print(time.time() - start)
result.show()
json_res = result.export()

for res, rnd in zip(json_res["pages"], rnds):
    orientation = res["orientation"]["value"]
    print(f"orientation: {orientation}, rnd: {rnd}")

Results:

orientation: -63, rnd: -63
orientation: -21, rnd: -21
orientation: -17, rnd: -17
orientation: -55, rnd: -55
orientation: 85, rnd: 85
orientation: -14, rnd: -14
orientation: -47, rnd: -47
orientation: -83, rnd: -83
orientation: -54, rnd: -54
orientation: 20, rnd: 20
orientation: -82, rnd: -82
orientation: 33, rnd: 33
orientation: -4, rnd: -4
orientation: 5, rnd: 5
orientation: 51, rnd: 51
orientation: 61, rnd: 61
orientation: 34, rnd: 34
orientation: 31, rnd: 31
orientation: -4, rnd: -4
orientation: -22, rnd: -22

Code Tensorflow:

import random
import time

import requests

from doctr.io import DocumentFile
from doctr.models import ocr_predictor
from doctr.utils.geometry import rotate_image

#bytes_data = requests.get(
#    "https://www.allianzdirect.de/dam/documents/home/Versicherungsbedingungen-08-2021.pdf"
#).content

start = time.time()
# doc = DocumentFile.from_pdf("/home/felix/Desktop/vitstr_paper.pdf")
#doc = DocumentFile.from_pdf(bytes_data)
# doc = DocumentFile.from_images(['/home/felix/Desktop/5.png']) # * 20)
doc = DocumentFile.from_pdf('/home/felix/Desktop/dansk.pdf') * 20
rnds = [random.randint(-90, 90) for _ in range(len(doc))]
# rnds = [random.choice([90, -90]) for _ in range(len(doc))]
doc = [rotate_image(page, rnd, expand=True) for page, rnd in zip(doc, rnds)]

model = ocr_predictor(
    det_arch="linknet_resnet18_rotation",
    reco_arch="parseq",
    pretrained=True,
    det_bs=8,
    reco_bs=1024,
    assume_straight_pages=False,
    straighten_pages=True,
    detect_orientation=True,
)#.cuda().half()  # uncomment if you have a GPU

result = model(doc)
print(time.time() - start)
result.show()
json_res = result.export()

for res, rnd in zip(json_res["pages"], rnds):
    orientation = res["orientation"]["value"]
    print(f"orientation: {orientation}, rnd: {rnd}")

Results:

orientation: 38, rnd: 38
orientation: -59, rnd: -59
orientation: 19, rnd: 19
orientation: -24, rnd: -24
orientation: -90, rnd: 90
orientation: -16, rnd: -16
orientation: -4, rnd: -4
orientation: -11, rnd: -11
orientation: 48, rnd: 48
orientation: -21, rnd: -21
orientation: -34, rnd: -34
orientation: 39, rnd: 39
orientation: -36, rnd: -36
orientation: 57, rnd: 57
orientation: -25, rnd: -25
orientation: 2, rnd: 2
orientation: 52, rnd: 52
orientation: 65, rnd: 65
orientation: 40, rnd: 40
orientation: 39, rnd: 39

Everything looks as expected on my side 😅

Are you sure that you are checked out correctly into my changes ?

(doctr-dev) felix@felix-GS66-Stealth-11UH:~/Desktop/doctr$ git branch
  benchmark-docs
  main
* orientation-seq-prototype
  tf-detection-augmentations

• marked the branch you need to checkout

[felixdittrich92]

Otherwise, please be patient. I think @odulcy-mindee can review the rest on Friday, so we can merge it next week at the latest.
Then you could pull the changes directly from main branch.

[denivic]

I may not have the correct version then. I apologize for the confusion and waste of time. I'll have to double check tomorrow and try to verify using the code above. I'll do a 100% clean re-install of DocTR and its dependencies. I'll let you know. Again, sorry. :)

[felixdittrich92]

I may not have the correct version then. I apologize for the confusion and waste of time. I'll have to double check tomorrow and try to verify using the code above. I'll do a 100% clean re-install of DocTR and its dependencies. I'll let you know. Again, sorry. :)

It's all ok :)

Ensure that you are on the correct branch : orientation-seq-prototype

[denivic]

@felixdittrich92 Alright so it turns out I did have the latest version installed, but something somewhere went wrong. I uninstalled and then reinstalled DocTR and all its dependencies to test once again and it works without issue now (besides small inconcistencies, but it's waaaaay better than my previous results still).

orientation: 41, rnd: -48
orientation: 53, rnd: 53
orientation: 69, rnd: 69
orientation: 78, rnd: 78
orientation: 41, rnd: 41
orientation: 41, rnd: 41
orientation: 54, rnd: 54
orientation: 75, rnd: 75
orientation: 77, rnd: 77
orientation: 63, rnd: 63

I noticed I had duplicate installations of both DocTR (same version though) and its dependencies (also same version). Does pip not detect that there's a package with the same version already installed?

Anyway, I'm looking forward to the next release. For now this is great though. Thank you so much for the help. :)

[felixT2K]

Normally it should 😅
Nice that it now works on your side also 👍🏼
The results will further improve with the new detection models :)

[denivic]

Hello @felixdittrich92

I've been watching #1336 closely and I'm happy that you've put so much work in. Appreciate it.

I was just wondering if there's an ETA on when 0.8 will release? 😄

[felixT2K]

Hey @denivic 👋🏼

Thanks for your kind words 😄

I depends a bit how fast we get the following ticket / runs done #1364
So personally i hope that we are done with it in january afterwards only a bit minor stuff left which should be not take more then 1 week and we can do the release

[felixT2K]

merged :)

[nehashinde-canopy]

I am getting an error while running docTR:
predictor = detection_predictor('db_resnet50_rotation', pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True)
error:
ValueError: unknown architecture 'db_resnet50_rotation'
I have installed pytorch version of doctr using pip install "python-doctr[torch]"
Kindly help.

[felixdittrich92]

db_resnet50_rotation is removed in v0.8.0 it is now db_resnet50

[nehashinde-canopy]

Is there no way I can detect the rotation angle in the image?

…

On Fri, Mar 1, 2024 at 3:38 PM Felix Dittrich ***@***.***> wrote: db_resnet50_rotation is removed in v0.8.0 it is now db_resnet50 — Reply to this email directly, view it on GitHub <#1283 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BGKCWN36SIBUG4QJR3NHQKTYWBHQ3AVCNFSM6AAAAAA3XEETMCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMNBRGIYTM> . You are receiving this because you commented.Message ID: ***@***.***>

-- This message is intended for the use of the addressee and may contain  information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in this message is strictly unauthorized and prohibited. If you have received this message in error, please notify the sender by reply e-mail and delete the message from your system.

[felixdittrich92]

That's possible with the ocr_predictor instance and detect_orientation=True :)

[nehashinde-canopy]

Thanks I'll try this. Is there a way I can get a key value kie extraction like: { Name: Neha, Dob: 26-03 } I get only "words" as a class when I implement KIE. I need keys and values from the document. Thanks, Neha

…

On Fri, 1 Mar, 2024, 7:25 pm Felix Dittrich, ***@***.***> wrote: That's possible with the ocr_predictor instance and detect_orientation=True :) — Reply to this email directly, view it on GitHub <#1283 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BGKCWNZUZNZTSTF66SF64C3YWCCGVAVCNFSM6AAAAAA3XEETMCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMNBTGQ2DC> . You are receiving this because you commented.Message ID: ***@***.***>

-- This message is intended for the use of the addressee and may contain  information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in this message is strictly unauthorized and prohibited. If you have received this message in error, please notify the sender by reply e-mail and delete the message from your system.

[Tomerbarm]

We hit this exact problem building a product-photo pipeline — OCR/vision models degrade badly on rotated input and most general solutions don't reliably tell you which rotation you're looking at (particularly 90 vs 270, a well-documented confusion — see RotBench, EACL 2026, arXiv:2508.13968). Ended up training a small dedicated model for just this sub-problem rather than relying on the main model to self-correct. Not trying to plug a product in an issue thread, just sharing in case the underlying approach (cardinal-rotation classifier + fine-angle regression, arbitrated) is useful prior art for anyone tackling this here. Happy to share more on the methodology if useful: https://doi.org/10.5281/zenodo.22975679