r/learnpython • u/AR3126 • 19h ago
Variability in langdetect output
Hi all,
I’m using Python to filter out rows that contains non-English text from a noisy 10,000-row Excel dataset, and the final count varies slightly between runs (for example, keeping 7,522 vs. 7,508 cells—a difference of less than 0.001% of total cells). Is that normal for noisy data, and is that level of variance acceptable for research?
Thanks all!
3
Upvotes
3
u/Brian 18h ago
It maybe depends on how you're doing it, but I'd kind of expect something like that to be deterministic, at least if using a locally running method. It might not perfectly detect the correct language every time, but I'd expect the same result from run to run unless the data or code itself changes.
It might vary if it's doing something like hitting a network service for language detection, but otherwise it may be something going wrong (eg. not handling errors correctly, or a race condition (if done in parallel), so may be worth debugging a bit.
1
u/egotherapy 17h ago
In my experience, when strings are really short, langdetect has problems detecting languages correctly. I'd expect that you've filtered out really short/long/nonsense/corrupt text, but if not, that may account for some of the variability.
2
u/lakseol 12h ago
The
langdetectpage in PyPi has this:You will have to experiment a bit to see if this helps in your case.