Images, video and voice

What AI can see, hear and generate beyond text. About 30 min, beginner level.

You'll be able to

• Explain how a multimodal model takes in an image or audio, and how that differs from generating one.
• Use an assistant to transcribe and organize handwritten or visual material, and find the errors it makes.
• Apply consent and labeling rules when you create or share AI-generated images, audio or video.

One model, many kinds of input

A multimodal model works with more than text. Many assistants can now read a photo, a chart, a screenshot or a scanned page, and some can listen to speech and talk back.

Here's roughly how an assistant reads an image. An image encoder, a neural network trained on large numbers of images, converts the picture into a sequence of vectors, often one for each small patch of the image. These are mapped into the same kind of numerical space as the model's text tokens. The language model then attends to the image vectors and your words together, which is how it can answer “What does this chart show?”

Audio works in a similar way, or it is first turned into text by a speech recognition model. Either way, the reply is still text generated one token at a time, unless a speech model reads it aloud.

The key idea: the model “sees” an image as a sequence of learned vectors placed alongside your words, not as a person sees it.

Reading images well, and where it slips

Assistants are often good at transcribing printed text and reasonably neat handwriting, describing a scene, explaining a diagram, identifying the type of chart, and turning a photographed table into rows and columns.

They are weaker at small or blurry text, messy handwriting, counting many similar objects, reading exact values off a chart, fine spatial detail, and specialist notation such as equations or chemical structures.

The real danger is how they fail. When a model can't make out a word, it tends to write a plausible word rather than flag the gap. A dropped minus sign or a misread name comes out in the same clean text as everything else, so the transcript looks trustworthy even when it isn't.

What helps:

• Photograph straight on, in good light, and crop to what you need.
• Ask it to mark unreadable words as [?] instead of guessing.
• Compare the output with the original, line by line.

The key idea: a clean-looking transcript can hide confident guesses.

Generating images, audio and video

Understanding media and generating it are different jobs, usually done by different models.

Many image generators use diffusion. The model starts from random noise and removes it step by step, guided by your text prompt, until an image matching the description emerges. Video models extend this idea across frames. Voice models turn text into speech, and some can imitate a particular person's voice from a short recording.

Generated media follows the patterns in its training data, not the facts of your subject. That means:

• It can fall back on stereotypes, such as who a “scientist” or a “nurse” looks like.
• Text inside images and fine details are often garbled.
• A generated diagram of a cell or a circuit can look authoritative and be wrong.

Legal questions about training data and copyright are still being worked out, so check the tool's terms and your course or unit rules before using generated media in your work.

The key idea: a generated image is a plausible picture, not a record of anything real.

Consent, labels and deepfakes

Realistic generated media of real people, often called deepfakes, can deceive, harass or damage someone's reputation. A few rules cover most situations:

• Consent first. Never create realistic images, video or a cloned voice of a real person without their permission. That includes classmates, colleagues and instructors, even as a joke.
• Label what you share. Say when an image, audio clip or video is AI-generated, in a caption or credit line, so no one mistakes it for a record of something real.
• Know the limits of provenance. Some tools attach content credentials (based on the C2PA standard) or invisible watermarks to what they generate. These help, but they can be stripped, so their absence proves nothing.

When you receive striking media, check where it came from and whether reliable sources report the same thing. AI detectors alone are not reliable enough to settle the question.

University conduct and academic integrity rules still apply when the content is AI-made.

The key idea: get consent, label it, and verify before you share.

From whiteboard to study sheet

A student photographs a page of handwritten statistics notes and sends it with this prompt:

“Transcribe this page exactly. Mark any word or symbol you can't read as [?]. Don't fix, add or explain anything. Then, separately, organize the transcription under headings with bullet points, using only what's on the page.”

Asking for two separate steps matters: first a faithful transcript, then the reorganized notes. It makes it much easier to see where the model changed something.

Checking it. The student lays the transcript beside the photo and marks every difference. The kinds of errors to look for:

• Misread words: a similar-looking word swapped in, such as “variation” for “variance”.
• Dropped or changed symbols: a formula's n−1 becoming n, or a missing square root.
• Added content: a definition or example that isn't on the page, which is most likely to slip into the organized version.
• Silent guesses: words it didn't mark [?] even though the handwriting is unclear.

They correct the study sheet, count each type of error, and keep the corrected version, not the raw output.

Try it

Your photo of the notes, the assistant's transcription and organized version, and a marked-up copy showing every error (misread words, changed symbols, added content), with a count of each type.