CLIP-CLOP & Arnheim: Neural Visual Grammars & Collage
Artist-in-the-loop evolutionary photomontage and hierarchical visual grammars guided by multimodal dual encoders—giving working artists transparent, object-level control over generative composition.
Why Compositional Grammars Instead of Opaque Pixels?
Before diffusion models came to dominate text-to-image generation, pixel-based GANs offered little structural transparency or step-by-step control for working visual artists. Named after perceptual psychologist Rudolf Arnheim (author of Art and Visual Perception), the Arnheim research stream—spanning Neural Visual Grammars (Arnheim 1, 2021) and CLIP-CLOP (Arnheim 3, 2022)—explored how compositional, object-based representations could be paired with pre-trained vision-language dual encoders (such as CLIP and ALIGN) to create transparent, manipulable generative art tools.
1. Arnheim 1: Neural Visual Grammars (2021)
Generative Art Using Neural Visual Grammars and Dual Encoders (arXiv:2105.00162) combined hierarchical neural Lindenmayer systems (L-systems) and differentiable vector/stroke rendering with large-scale image-text dual encoders.
Rather than optimizing raw RGB pixels, the system evolved populations of structured visual abstractions driven by natural-language prompts—demonstrating how compositional recursive grammars yield interpretable, non-photorealistic visual styles with coherent global structure and self-similar detail.
2. CLIP-CLOP: CLIP-Guided Collage & Photomontage (ICCC 2022)
Inspired by Dadaist photomontage (Hannah Höch, Raoul Hausmann) and Pop Art collage (Richard Hamilton, Eduardo Paolozzi), CLIP-CLOP (arXiv:2205.03146) replaces opaque pixel synthesis with a differentiable and evolutionary collage generator:
- Spatial Transformer Patch Canvas: A set of segmented image patches (curated by the artist from personal photographs or sampled from transparent PNG libraries) are positioned, scaled, rotated, perspective-warped, recolored, and layered onto a canvas via differentiable spatial transformer networks.
- Hybrid Gradient + Evolutionary Optimization: Gradient ascent optimizes continuous patch coordinates and color shifts to maximize semantic similarity with a text prompt via CLIP embeddings, while an outer evolutionary population loop swaps, mutates, and recombines discrete patch selections so compositions don't get trapped in local minima.
- Artist-in-the-Loop Co-Creation: Because every element on the canvas remains a discrete, editable object with explicit coordinates and alpha transparency, human artists can curate custom patch libraries, lock or reposition individual collage pieces mid-run, and iteratively steer the composition in open-source Google Colab notebooks.
3. Embodied in Hardware: Vis Plastica
To bring slow, evolutionary generative art off the cloud workstation and onto the living-room wall, Dylan built Vis Plastica: a dedicated low-end single-board computer mounted inside a physical picture frame that runs neural visual grammar and collage evolution locally 24/7, slowly mutating a painting before the viewer's eyes over hours and days.