OnOff: Bridging Online and Offline Handwriting via Differentiable Physical Rendering

Seonmi Park¹, Seunghyun Shin¹, Vihaan Misra², Dongmin Shin³,
Ukcheol Shin^4*, Jean Oh², Hae-Gon Jeon^3*

¹GIST ²CMU ³Yonsei University ⁴KENTECH

^*Corresponding authors

ECCV 2026

arXiv Code (Coming Soon!) Data (Coming Soon!)

OnOff unifies online and offline handwriting generation via a differentiable physical brush renderer enabling trajectory estimation, image rendering, image refinement, and direct robot execution within a single framework.

Framework

We integrate online and offline handwriting generation into a unified, physics-based differentiable rendering framework. By disentangling brush dynamics from pen trajectories, our approach enables highly controllable and realistic stylistic synthesis.

Fig. 1. Overview of the proposed joint online-offline handwriting framework.

Online Stroke Generator G: Predicts a word-level stroke trajectory S from style references I_sty and text T
Brush Prameter Observer O: Estimates brush parameters θ from references I_sty (Controllable & Explainable)
Differentiable Brush Renderer R: Maps the trajectory S and brush parameters θ to an image I_rend
Zero-shot Image Enhancer E: Adds realistic paper textures and visual refinements using diffusion

Differentiable Brush Renderer

Six physically-interpretable parameters θ = { w_base, k_spread, ρ_ink, σ_sharp, p_min, p_max } control brush footprint geometry, ink transfer, and stroke composition, all differentiable.

Fig. 2. Conceptual overview of the proposed brush model.

Fig. 3. Effect of brush parameters.

Online-Offline Paired Dataset

Using our renderer, we build a synthetic paired dataset D_syn from existing online corpora (IAM-OnDB, CASIA-OLHWDB), composited onto diverse real-world backgrounds.

Online-Offline paired dataset (example 2).

Fig. 4. Paired samples generated by our differentiable renderer, with diverse backgrounds, styles, and word lengths.

Creation: Renders real online trajectories into realistic offline images using randomized brush parameters and paper backgrounds
Purpose: Provides fully paired supervision (T, θ, S, I_syn) to train the unified online-offline framework

Qualitative Comparison

Combining our differentiable renderer with existing offline baselines (DiffPen, One-DM, Emuru, VATr++) improves both structural consistency and visual realism.

Fig. 5. Qualitative comparison of offline handwriting generation results.

Application: Robotic Calligraphy

Predicted stroke trajectories and brush parameters are executed on a Ufactory Lite 6 robot: the (x, y) path drives horizontal motion, and the pressure proxy modulates the vertical position z.

Results across writing tools

Robot writing execution across writing instruments.

Fig. 6. Robot writing execution results.

Fig. 7. Robot writing demonstrations.

Sentence-Level Application

Word-level trajectories compose into sentences via fixed spatial offsets, scalable to both offline generation and robotic execution.

Fig. 8. Sentence-level application. (A) rendered image, (B) offline image, (C) robot execution results.

BibTeX

@inproceedings{park2026onoff,
  title     = {OnOff: Bridging Online and Offline Handwriting via Differentiable Physical Rendering},
  author    = {Park, Seonmi and Shin, Seunghyun and Misra, Vihaan and Shin, Dongmin
               and Shin, Ukcheol and Oh, Jean and Jeon, Hae-Gon},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}