Research
What makes it feel tangible on iPhone (iPad after)
Question: on a phone with no stylus and no pressure sensing, what makes paint feel like a physical substance under your finger?
iPhone first is not a compromise
- It's a held object. You tilt a phone constantly without thinking. Gyro relighting is natural on a phone and slightly effortful on an iPad lying on a desk.
- It has the best haptics in consumer electronics (Taptic Engine + Core Haptics), in the hand that's holding it.
- The TrueDepth camera faces the viewer, and therefore the room light falling on the painting (see Depth and light).
- It's always with you. The 1841 collapsible paint tube is usually credited with enabling plein-air Impressionism — Renoir: "Without paints in tubes, there would have been no Cézanne, no Monet, no Sisley or Pissarro." (Honest caveat: plein-air painting predates the tube, and portable easels and paint boxes mattered as much.) The phone is the next portable format.
- A small canvas makes the simulation easier, not harder. 1–2k² with a 4-pigment Zorn palette plus height, wetness and tangent is ≈ 12 B/texel ≈ 50 MB. That's a fraction of the iPad budget.
- You can pass it across a table. Handing someone your phone and watching them tilt it is the share loop, and it only exists on a phone.
The honest constraints
- No pressure. 3D Touch was removed from iPhone 11 onward, and
UITouch.forcewith it.UITouch.majorRadiusis the usual workaround, but it gives only 2–3 useful levels, and it follows finger angle more than force. - No Pencil, no hover, no tilt or azimuth.
- The finger hides the mark it's making.
Inputs, reframed — design for the finger, not against it
- Contact radius as contact shape, not pressure. Fingertip versus flat pad is exactly how painters use fingers: tip for accents, pad for smoothing. Titian did this in his late works — Palma Giovane (via Boschini, 1674) says he finished "more with his fingers than his brushes." Map radius to brush width and flatness.
- Speed and dwell as load. Slow strokes lay more paint; a stationary press pushes paint in.
- Gyro is light, never brush. Keep the channels clean. Tilt must never change the mark.
- Occlusion: test a small offset "lead" of the contact point, and a live swatch of the brush load near the finger.
Outputs — the tangible stack
- Continuous haptic texture (native only). A Core Haptics continuous player with dynamic parameters updated every frame: - intensity ← drag resistance (viscosity × paint load × speed) - sharpness ← surface: canvas tooth is sharp and grainy; thick wet paint is soft and dull. Sharpness maps to frequency only on continuous events. - transients when the finger crosses a ridge (height-gradient threshold), and a crisp tick when the knife catches a dry ridge. - modulate intensity by the tooth heightfield sampled under the finger at finger speed, so you feel the weave at the right spatial frequency. Test what the actuator (resonant around 160 Hz) can actually render.
No painting app we found does this. It's the single most tangible thing an iPhone can do. The web cannot do it (iOS Safari has no Vibration API), so it's a native experiment. 2. Gyro relight, with light and eye fixed in the room so both rotate relative to the canvas as you tilt. More correct than "light follows the tilt." 3. Face-tracked view (optional, opt-in). 4. HDR gloss — wet highlights above paper white on XDR screens. 5. Sound, very quiet and off by default. Bristles on tooth, a knife scrape. 6. Cross-modal coherence beats intensity. The haptic tick has to land on the same frame the highlight breaks. Keep latency aligned across channels.
Product shape this implies
- iPhone = the pocket study. Alla prima, one small panel, 10–20 minutes — the oil sketch on a board, like Constable's outdoor sketches or a plein-air pochade box.
- iPad = the studio painting, later. Apple Pencil Pro adds real pressure, tilt, barrel roll and haptics in the pen.
- One document format across both, so a pocket study can grow into a studio painting.
What to test
| Test | Pass if |
|---|---|
Web lab (this repo, lab/) on a real iPhone: gyro relight + finger paint |
People tilt without being told to, and the paint reads as raised |
| Native haptic canvas: drag a finger over a static heightfield with texture haptics on/off | Blind, people can tell canvas from wet paint from dry ridge by feel |
| majorRadius mapped to brush width | Tip vs pad is controllable on purpose within a minute |
Sources
- Core Haptics design notes — Daniel Büttner · WWDC 2019 · Expanding the Sensory Experience with Core Haptics · Donny Wals on Core Haptics
- Force Touch — Wikipedia (3D Touch removed) · use-the-force-touch (majorRadius levels) · Pseudo3DTouch
- John Goffe Rand invents paint tubes — History Today · Smithsonian: Never underestimate the power of a paint tube
- Titian's technique — ARTEnet · Artibus et Historiae