r/deeplearning • u/Safe-Escape197 • 21h ago
80 downloads on my tiny vision model so i made v1.0, now it lets claude “watch” videos (36M params, runs on laptop cpu)
so few days back i posted my tiny vision model that i built just for learning (v0.1), and 80 people downloaded it on huggingface which honestly shocked me lol. people in comments gave really good feedback so i actually worked on all of it, and now v1.0 is out.
main thing: claude (and many other assistants) can look at images but cant open a video file. so tiny-vision runs locally next to it, breaks the video into scenes and gives the assistant a scene by scene breakdown. then you just ask “what happens in this video” and it actually answers. nothing gets uploaded, all on your machine. i tested it with a 10-15 min youtube video and it still works fine.
works through MCP so it runs in claude desktop and claude code (tested both), and should work in cursor, vs code, lm studio etc (havent tested those yet). if your app dont have MCP you can just pipe the text into ollama or paste it anywhere.
what changed from v0.1 (thanks to the comments):
- someone said the “a man in a blue shirt” thing is probably the encoder not the decoder. i tested it and yaar they were right. my from-scratch encoder only saw 29k images, so i switched to a frozen pretrained DINOv2-small (meta’s) and trained the adapter + decoder myself (14M params) on COCO
- it picks frames on scene changes now instead of one every 2 sec, like someone suggested. so the timestamps land on the actual cuts
- real benchmarks this time, not vibes
numbers (COCO test):
- v0.1: CIDEr 0.06 💀
- v1.0: CIDEr 1.16
vs florence-2-base (231M) on the same 1000 images:
- ~89% of its caption score
- 6x smaller
- 8x faster on gpu
- 75x faster on cpu (54ms per image vs 4 sec)
not saying its better than florence, its not. it just does a lot with very little.
honest stuff:
- encoder isnt mine anymore, its DINOv2. i tried my own and it just cant compete without millions of images
- no audio yet, so it describes what it sees, not what is said. speech is coming next
- captions are COCO style, so no OCR, no names, no counting
- and like last time, i took a lot of help from claude to build this, so its not entirely mine
previous post: https://www.reddit.com/r/computervision/comments/1xyuaia/built_a_tiny_vision_model_from_scratch_just_for/
code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision
anything you want me to try next? still learning so all feedback welcome




