The original Vision Mouse used a fixed threshold to detect blinks, which worked fine for me, but broke the moment anyone else with different eye shape or lighting tried it. Vision Mouse Advanced fixes that with a short calibration phase that learns your blink before the cursor ever starts moving. Here's how the code actually works, from facial landmarks to click.

What Changed From V1

  • 🧠 Smart calibration, detects your personal blink threshold instead of using a hardcoded one.
  • 🖱️ Smoother real-time cursor control using eye landmark interpolation.
  • 👁️ Click with a natural blink, no hardware, no keyboard shortcuts.
  • 📷 Live webcam feedback showing tracked landmarks while it runs.

The stack is Python, OpenCV for the video loop, MediaPipe's Face Mesh for landmark detection, and PyAutoGUI to actually move the OS cursor and click.

Setting Up Face Tracking

MediaPipe's Face Mesh does the heavy lifting: point a webcam at a face and it returns hundreds of normalized landmark coordinates every frame, no training required.

cam = cv2.VideoCapture(0)
face_mesh = mp.solutions.face_mesh.FaceMesh(refine_landmarks=True)
screen_w, screen_h = pyautogui.size()

refine_landmarks=True is the important flag here, it turns on MediaPipe's iris refinement, which is what gives access to the extra eye/iris landmarks (indices 474–478) used later for cursor tracking. Without it, eye tracking would be far less precise.

Measuring an Eye Blink

A blink is detected by measuring the vertical distance between two landmarks on the left eye, top lid and bottom lid. When the eye is open, that gap is relatively large; when it closes, the gap shrinks toward zero.

def get_eye_diff(landmarks, frame_h):
    left = [landmarks[145], landmarks[159]]
    return abs(left[0].y - left[1].y)

Landmarks 145 and 159 are two fixed points on the left eyelid in MediaPipe's face mesh topology. This function is reused in both the calibration phase and the main loop, it's the single source of truth for "how open is the eye right now."

The Calibration Phase

Instead of guessing a blink threshold, the program asks the user to blink five times and records the eye-diff value at each one:

blink_diffs = []
required_blinks = 5

while len(blink_diffs) < required_blinks:
    _, frame = cam.read()
    frame = cv2.flip(frame, 1)
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    output = face_mesh.process(rgb_frame)

    if output.multi_face_landmarks:
        landmarks = output.multi_face_landmarks[0].landmark
        diff = get_eye_diff(landmarks, frame_h)

        if diff < 0.014:
            blink_diffs.append(diff)
            print(f"Blink {len(blink_diffs)} recorded.")
            time.sleep(1)  # wait for eye to open again

# ...
blink_threshold = np.mean(blink_diffs) + 0.002

0.014 is a conservative starting cutoff, low enough that it only fires on an actual full blink, not just a squint. Once five samples are collected, the personal threshold is set slightly above their average (+0.002), which gives a small buffer so tiny natural fluctuations in eye-diff don't accidentally register as a blink once the real tracking starts. This one calibration step is what makes the system work across different people instead of just the one it was tuned on.

Moving the Cursor

Once calibrated, the main loop reads iris landmarks and maps their position directly onto the screen:

for id, landmark in enumerate(landmarks[474:478]):
    x = int(landmark.x * frame_w)
    y = int(landmark.y * frame_h)
    if id == 1:
        norm_x = np.clip(landmark.x, 0.1, 0.9)
        norm_y = np.clip(landmark.y, 0.1, 0.9)
        screen_x = np.interp(norm_x, [0.1, 0.9], [0, screen_w])
        screen_y = np.interp(norm_y, [0.1, 0.9], [0, screen_h])
        pyautogui.moveTo(screen_x, screen_y)

Landmarks 474–478 are the refined iris points unlocked earlier by refine_landmarks=True. The clip to [0.1, 0.9] before interpolating is a deliberate choice: MediaPipe's normalized coordinates run 0 to 1 across the frame, but the outer 10% on each edge is where iris tracking gets noisiest (people rarely look at the extreme corners of a webcam anyway). Clipping that range and then interpolating to the full screen width/height means small, controlled eye movements map to the whole screen instead of just the usable center of the frame.

Clicking With a Blink

Detecting the click itself reuses get_eye_diff from calibration, now compared against the personalized threshold, with a simple state flag to stop it from firing repeatedly while the eye stays shut:

if diff < blink_threshold and not blinked:
    blinked = True
    pyautogui.click()
    print("Blink")
    time.sleep(0.3)
elif diff >= blink_threshold:
    blinked = False

The blinked flag is what turns a blink into a single discrete click instead of a rapid-fire burst of clicks for every frame the eye happens to be below threshold. It only resets once the eye reopens (diff >= blink_threshold), so one physical blink reliably produces exactly one click.

Why the Calibration Step Matters

The lazy version of this project would hardcode a single blink threshold and call it done, and it would work great in a demo and fail for basically everyone else. Baking in a five-second calibration step is a small amount of extra code for a much more robust result: the system adapts to each user's actual eye shape, camera angle, and lighting instead of assuming everyone blinks identically.

Try It Yourself

Full code is open-source under the MIT License on GitHub: Sharkyy-eng/vision-mouse-advanced.

Run python main.py, blink five times when prompted to calibrate, then control the cursor with your gaze and click with a blink. A well-lit room and a steady face during calibration make a noticeable difference in tracking quality.