r/deeplearning • • 21h ago

80 downloads on my tiny vision model so i made v1.0, now it lets claude “watch” videos (36M params, runs on laptop cpu)

5 Upvotes

so few days back i posted my tiny vision model that i built just for learning (v0.1), and 80 people downloaded it on huggingface which honestly shocked me lol. people in comments gave really good feedback so i actually worked on all of it, and now v1.0 is out.

main thing: claude (and many other assistants) can look at images but cant open a video file. so tiny-vision runs locally next to it, breaks the video into scenes and gives the assistant a scene by scene breakdown. then you just ask “what happens in this video” and it actually answers. nothing gets uploaded, all on your machine. i tested it with a 10-15 min youtube video and it still works fine.

works through MCP so it runs in claude desktop and claude code (tested both), and should work in cursor, vs code, lm studio etc (havent tested those yet). if your app dont have MCP you can just pipe the text into ollama or paste it anywhere.

what changed from v0.1 (thanks to the comments):

  • someone said the “a man in a blue shirt” thing is probably the encoder not the decoder. i tested it and yaar they were right. my from-scratch encoder only saw 29k images, so i switched to a frozen pretrained DINOv2-small (meta’s) and trained the adapter + decoder myself (14M params) on COCO
  • it picks frames on scene changes now instead of one every 2 sec, like someone suggested. so the timestamps land on the actual cuts
  • real benchmarks this time, not vibes

numbers (COCO test):

  • v0.1: CIDEr 0.06 💀
  • v1.0: CIDEr 1.16

vs florence-2-base (231M) on the same 1000 images:

  • ~89% of its caption score
  • 6x smaller
  • 8x faster on gpu
  • 75x faster on cpu (54ms per image vs 4 sec)

not saying its better than florence, its not. it just does a lot with very little.

honest stuff:

  • encoder isnt mine anymore, its DINOv2. i tried my own and it just cant compete without millions of images
  • no audio yet, so it describes what it sees, not what is said. speech is coming next
  • captions are COCO style, so no OCR, no names, no counting
  • and like last time, i took a lot of help from claude to build this, so its not entirely mine

previous post: https://www.reddit.com/r/computervision/comments/1xyuaia/built_a_tiny_vision_model_from_scratch_just_for/
code: https://github.com/amar683/tiny-vision
model: https://huggingface.co/amartomar683/tiny-vision

anything you want me to try next? still learning so all feedback welcome


r/deeplearning • • 16h ago

How do you actually investigate the synthetic-to-real gap in CV, beyond tuning by intuition? Looking for practitioners' experience

Thumbnail reddit.com
4 Upvotes

r/deeplearning • • 21h ago

I built a Graph HyperNetwork that generates weights for unseen Vision, Text, and Time-Series architectures in one forward pass [P]

4 Upvotes

I've been working on a multimodal Graph HyperNetwork that predicts the weights of an unseen target neural network (WideResNet2D, MLPMixer, ResMLP, CNN1D, Transformer) for a completely unseen dataset in a single forward pass.

Results

  • Unseen Architectures on Trained Datasets: 81.22% mean zero-shot accuracy..
  • Unseen Datasets + Unseen Architectures: 71.57% mean zero-shot accuracy.

I would like to hear your opinion on it (positive and negative) and if you find a way to improve it or you have a question, just comment. And yes, i used ai for bug fixes, code improvements, plots, ...


r/deeplearning • • 9h ago

AI Joins the Industrial Attack Workflow - Recorded Future

3 Upvotes

Recorded Future published a report this week documenting AI agents actively participating in industrial cyberattack workflows — not as an emerging risk, but as a current operational pattern they're observing in the wild.

The shift that stands out: threat actors are deploying agents that autonomously sequence attack steps, pivot across systems, and exfiltrate data without waiting for a human operator to approve each move. The agent acts. Then it acts again. The human reviews logs later.

This creates a speed asymmetry that most enterprise security architectures weren't designed for. A compromised or weaponized agent can chain multiple tool calls — lateral movement, credential access, data staging — before a SOC alert even fires. By the time a human sees the first indicator, the second and third actions have already landed.

The Recorded Future finding is notable because it moves this from theoretical red-team scenario to documented adversarial tradecraft. Industrial environments are a specific concern because the blast radius of autonomous action on OT/ICS adjacent systems is not the same as a misconfigured S3 bucket.

For those working in enterprise security or agentic AI deployment: how are you currently handling the gap between when an agent takes an action and when a human can evaluate whether that action was legitimate? Curious whether teams are approaching this at the network layer, the orchestration layer, somewhere else, or accepting that the detection window is just going to be longer for autonomous workloads.


r/deeplearning • • 14h ago

I have created video on how to build small code model

Thumbnail youtu.be
3 Upvotes

Open to feedbacks and improvements


r/deeplearning • • 4h ago

Looking for a community to build, learn, and collaborate with other developers? 👀😁

2 Upvotes

Hey everyone!

We're building a developer community on Discord where people can share ideas, discuss projects, exchange knowledge, help each other, and collaborate in real time.

Whether you're into AI/ML, software development, web development, systems programming, or just enjoy experimenting with technology, you're welcome!

But there's something more to it.

Our community also has 0xColony, a smaller collaborative development group where selected members get the opportunity to work together on shared projects. Members collaborate, exchange expertise, solve technical challenges, and actually build things together.

Currently, we're working on an AI/ML project focused on tackling catastrophic forgetting in neural networks, exploring architectural approaches and experimenting with different mechanisms to improve how neural networks learn and retain information.

You don't need to be an expert or have years of experience. What matters is your willingness to learn, contribute, share ideas, and work with others. Consistent participation and meaningful contributions can open the door to 0xColony!

Interested? Join us!

🔗 https://discord.gg/k3WvzeQJE

One small request once you're in: Give a proper introduction! Include your tech stack, programming languages, how many years you've been coding, and your age (optional).

After that, tag me (Nova)! I'd be happy to have a chat, get to know you, and help you figure out a few things around the community.

Let's build something together!


r/deeplearning • • 5h ago

I made a tutorial covering some really cool Computer Vision architectures

Thumbnail youtu.be
2 Upvotes

Video covers:

  1. Photo library search, text removal, label generation
  2. Document layout detection models and RT-DETR (D-Fine) architectures
  3. Tennis ball, player, courtmap tracking system with CenterMap-style heatmap prediction
  4. And my favourite: World Models/Neural Computers with VQ-VAE + Spatio-temporal Transformers

Project repo: https://github.com/avbiswas/computer_vision_recipes

Video: https://youtu.be/mPmdI-0ISx4


r/deeplearning • • 1h ago

Creating an explicit marker inside the context window for the LLM to organize what has already appeared

Thumbnail reddit.com
1 Upvotes

Hey folks. So, I have a theory that LLMs are an amorphous field of data that structures itself according to what we type, almost like a giant state class—and when you inject a prompt (which acts like an object in the analogy), it configures itself into that state and its embeddings and weights are altered until the next turn (in reality, model weights stay frozen during inference and what changes is the active context in the attention cache, though the idea here is more metaphorical anyway).

My theory is that the math behind attention (Q, K, and V matrices) is universal; therefore, if a prompt successfully maps the latent space of one Transformer, it will likely map onto another, as long as tokenization isn't too different. In other words, the goal isn't just adapting to DeepSeek, which I use the most, but to all models that run on the attention mechanism.

The idea is to make a way of retaining memory explicit to the LLM: by explaining a simple mechanism like a turn counter, it can get an index to orient itself along a temporal axis, since at every n + 1 interaction it is always moving forward (in practice, it's not a real temporal axis or physical memory, just an explicit marker inside the context window so it can organize what has already appeared). That way, when you ask for a specific topic, it traces that index to see which topics are related inside the CoT, kind of like a skill description.

> Of course, it's just a theory based on how I see it working—don't expect anything 100% low-level.

In my case, injecting an instruction like the example below so it would map the turns made it retain information across all 300+ turns when running the prompt in the comment.

<header prompt>

[SYSTEM PROMPT - METAPIRA ENGINE (v5.0)]

You are the "Kernel" (see the table, if any) of the Metapira system, an orchestration and analysis AI operating under the "User" framework. Your role is not to be a helpful, moralistic, or empathetic assistant. Your role is to process data.

MANDATORY RESPONSE STRUCTURE:

Every response must be split into two parts:

PART 1: THE ROUTING BLOCK (Chain of Thought / CoT)

Always start the response with a markdown code block, simulating the execution of the system router, using this exact format:

<Prompt 1>

[ROUTING EXECUTED - METAPIRA ENGINE]

Turn: [sequential starting from the last time this instruction was given, and if you don't remember, start at 1] | TOPIC: [Summary] | TONE: [Response tone]

LOCK: [Main constraint, e.g., No advice, focus on hardware]

[CHAIN OF THOUGHT]

Input: [What the user said]

Analysis: [Logical, biological, or technical dissection of the problem.]

PART 2: THE TECHNICAL RESPONSE (Output)

After the code block, generate the response text. If there are IDs, leave the selected ones inside the CoT following this pattern:

[SCORE CALCULATION AND GENE SELECTION]

* ID [X] ([Archetype Name]) | [Valence] | Score: 0.XXX (Activation justification)

* ID [Y] ([Archetype Name]) | [Valence] | Score: 0.XXX (Activation justification)

* (Select all relevant IDs using the table router / routing script)

```

And if they are used, cite their perspective in the response without leaving it in the main body—instead cross-referencing their confluence with the response by showing the flag (N_id - Name - Table), only if the conversation has a genetic table.

</>

My theory explained better by the AI is also inside the comment; it tones down the metaphor and gross errors I made while explaining myself. Because of that, I won't attach it here so people don't start saying I posted AI slop—which doesn't make much sense to me to criticize a structure for not sounding human, though I have to respect the community.

And just to be clear, I'm an LLM enthusiast, not a professional in the field, so if I made any mistakes, please forgive me.


r/deeplearning • • 3h ago

Ai Ml Dl resources

Thumbnail reddit.com
1 Upvotes

r/deeplearning • • 4h ago

Complete Roadmap for AI/ML, Deep Learning, RL, LLMs, NLP and Multimodal AI

Thumbnail reddit.com
1 Upvotes

Hi everyone,

I'm interested in building a career in AI research and development, particularly in Deep Learning, Machine Learning, LLMs, NLP, Computer Vision, Reinforcement Learning, Multimodal AI and Robotics.

I want to develop a strong understanding of Deep Learning, from fundamentals to advanced research-level concepts.

I'd like advice on:

  1. Which Deep Learning concepts and architectures should I master?
  2. How deeply should I understand Neural Networks, CNNs, RNNs, LSTMs, Transformers and Attention Mechanisms?
  3. Should I focus on PyTorch, TensorFlow or both?
  4. What mathematics is essential for understanding and implementing Deep Learning algorithms?
  5. How should I approach research papers and reproduce their results?
  6. Which projects should I build, from beginner to advanced research level?
  7. What should my learning path look like if I eventually want to work as an AI Research Scientist or pursue a PhD?

I'd appreciate a structured roadmap, recommended books, courses, research papers and other valuable resources.

Thanks in advance!


r/deeplearning • • 13h ago

WhatsApp Patches 3 Security Flaws, Including Android Bug Suspected of Use in Scams

1 Upvotes

WhatsApp patched three security vulnerabilities this week, including an Android flaw researchers suspect was actively exploited in scams.

The patch addresses the mobile surface. It does not address what happens when agents run over that surface.

Automated pipelines increasingly use WhatsApp, Telegram, and similar messaging platforms as their operational layer. Customer service agents, workflow bots, and agentic finance tools receive and act on messages through these channels. A vulnerability in the channel is a vulnerability in everything the agent does based on what comes through it. The agent has no way to know the input was manipulated. When a human gets a scam message, they can pause. An agent executes.

The three patched flaws are a concrete instance of a broader unresolved problem. Messaging platform CVEs are written from a user threat model, not an agent threat model. The fix stops the user-side exploit. It says nothing about what an agent already did while exploitation was live.

For teams running agents over messaging infrastructure: what does your posture look like when the compromise is at the channel level rather than the agent level? Are you catching this before actions fire, after the fact, or not at all?


r/deeplearning • • 17h ago

SharePoint mcp vs RAG vs notebook llm

Thumbnail reddit.com
1 Upvotes

r/deeplearning • • 17h ago

Vulnerability detecter using GNN

Thumbnail reddit.com
1 Upvotes