Hidden Unicode Characters: How to Detect and Remove Them
Short answer: To find hidden Unicode characters, paste the suspicious text or code into the Hidden Unicode Detector. It lists every zero-width, direction-control, tag, variation-selector, unusual-space, control and look-alike character, shows them as visible labels, decodes hidden messages written in tag characters, and lets you remove what you choose. Everything runs in your browser.
What hidden Unicode is
Unicode contains thousands of characters that are invisible or that change how text is displayed. Two pieces of text can look identical on screen and still be different. Most of these characters have legitimate uses. A few are used in attacks.
Why it matters now
- Source code that reads one way and runs another. The Trojan Source research (CVE-2021-42574) showed that bidirectional control characters can reorder how code is displayed, so a reviewer sees harmless code while the compiler sees something else. A related issue (CVE-2021-42694) covers look-alike letters in identifiers.
- Instructions hidden from humans but readable by AI. Unicode tag characters map to ASCII but render as nothing. Text can carry an invisible message that a language model may read and a human reviewer cannot see. This is often called ASCII smuggling, and it is a concern for prompt injection, AI coding assistants and rules/config files that an assistant loads (see AGENTS.md and MCP config security).
- Phishing and filter evasion. Zero-width characters and look-alike letters can split a keyword or fake a domain so filters and eyes miss it.
- Bugs. A stray zero-width space or non-breaking space inside a YAML key, a password, a command or a CSV header can cause errors that look impossible.
A finding is not proof of an attack. Many of these characters are normal in everyday text.
What the detector checks
| Category | Examples | Why it matters |
|---|---|---|
| Zero-width characters | U+200B to U+200D, U+2060, U+FEFF | Invisible text, split keywords, stray BOM |
| Bidirectional controls | U+202A to U+202E, U+2066 to U+2069, U+200E, U+200F, U+061C | Trojan Source; reordering displayed text |
| Unicode tag characters | U+E0000 to U+E007F | Hidden ASCII messages; the tool decodes U+E0020–U+E007E |
| Variation selectors | U+FE00 to U+FE0F, U+E0100 to U+E01EF | Normal in emoji; can also hide data |
| Unusual spaces | no-break space, U+2000 to U+200A, U+202F, U+205F, U+3000 | Breaks comparisons, commands and parsers |
| Control characters | C0 and C1 controls (except tab, line feed, carriage return) | Terminal and parsing surprises |
| Look-alike letters | Cyrillic and Greek letters that resemble Latin, flagged inside otherwise-Latin words | Fake names and identifiers (pаypal with a Cyrillic "а") |
How to find hidden Unicode characters with the Hidden Unicode Detector
- Open the Hidden Unicode Detector.
- Paste the text or code, or click Example to see a sample. The text is analysed as plain text and is never rendered as HTML.
- Read the What was found summary: a count per category, with a "Clean" row when nothing was found.
- If a hidden message is found in tag characters, the tool shows the decoded text above the results.
- Check Text with visible markers, where each hidden character is replaced by a visible label so you can see exactly where it sits.
- Use the Findings table for position, line and column, code point, name and category.
- To clean the text, tick the categories you want to remove in the table. Hidden, tag, direction and control characters are removed, unusual spaces become a normal space, and look-alike letters become their Latin counterpart. The summary shows the counts before you copy the Cleaned text.
Nothing is changed unless you tick a box.
When hidden characters are normal
- Zero-width joiners and non-joiners are part of emoji sequences and of correct spelling in Persian, Urdu, Arabic and several Indic scripts. Removing them can change how text looks or even what it means.
- Variation selectors are used by emoji and some CJK characters.
- Direction marks are normal in mixed left-to-right and right-to-left text.
- A byte order mark (U+FEFF) at the very start of a file is common.
For that reason, review before cleaning. Removing joiners or variation selectors can change how emoji and some scripts display.
Practical places to check
- Code review: before merging code from an unknown contributor or an AI assistant (see insecure patterns in AI-generated code), scan new files and any config that tells an assistant how to behave.
- Copied commands: before running a command copied from a web page, check it. Pair this with the Shell Command Risk Explainer (see Is this shell command safe?).
- Prompts and documents fed to an AI: text pasted from the web or an email may carry invisible instructions.
- "Impossible" parse errors: a YAML, JSON or CSV file that looks right but fails (YAML errors guide).
- Account names, URLs and package names that look right but do not match.
Limits to know
- The look-alike check uses a short curated list of Cyrillic and Greek letters. It does not catch other scripts, mathematical letters, full-width forms, or whole words made only of non-Latin letters.
- It cannot catch text that is misleading without special characters, such as a similar-looking name or a misspelled package. For package names, use the Package Checker (see AI-hallucinated packages).
- Characters added or removed before you paste are invisible to the tool. The browser or the page you copied from may already have changed the text.
- Up to 2,000,000 characters are analysed. The marked-text view shows the first 50,000 characters and labels up to 1,000 findings in them, and the findings table lists the first 1,000. The counts always cover the whole text.
- Detecting is not the same as defending. Production systems should also normalise and sanitise input where it enters.
Frequently asked questions
How do I find hidden characters in text? Paste it into the Hidden Unicode Detector and read the findings.
How do I remove a zero-width space? Tick the hidden-characters category in the detector and copy the cleaned text.
What is the Trojan Source attack? Using bidirectional control characters so source code displays differently from how it is parsed.
What are Unicode tag characters used for? Originally for language tagging. Today they are best known for hiding ASCII text in plain sight.
Is my text uploaded? No. It is analysed on your device and never sent anywhere.
Related tools and guides
- Hex Viewer to inspect raw bytes
- ASCII Table
- Text Diff Checker to compare two "identical" strings
- String Escape & Unescape
- HTML Entity Encoder & Decoder
- URL encoding explained for percent-escapes that hide what a link really says
- AI Code Risk Linter and Secret Scanner