VLM Explainer: From Patches to Phrases

Wed, 01 Jan 2025 00:00:00 +0000

BLIP generates fluent captions but gives no account of why. The question this project answers: do the image regions the model highlights actually cause the caption, or do they just correlate with it?

Built Grad-CAM token attribution to identify which image regions influenced specific caption words. Validated through perturbation testing: masking the dog region changed “a family walking with their dog” to “a family walking on the beach.” Causal influence, not correlation, is the standard.

Added CLIP alignment scoring to cross-verify image-text grounding. Interactive Streamlit interface with 2-click region masking and layer-evolution visualization across shallow-to-deep BLIP representations.

Candidate Matcher

Mon, 01 Jan 2024 00:00:00 +0000

Keyword overlap misses semantic alignment. A resume can match a job description on surface terms and still be irrelevant.

Built ATS-style ranking using MiniLM embeddings and cosine similarity. LLM-generated summaries explain the match rather than just scoring it. Robust multi-format parsing pipeline for PDF, DOCX, and TXT files with fallback extraction logic. Deployed on Streamlit Cloud with Gemma 2B for summarization and Flan-T5 as a CPU-compatible fallback.

Streamlit | Jay Polra

VLM Explainer: From Patches to Phrases

Candidate Matcher