Language-Driven Artistic Style Transfer

Tsu-Jui Fu1   Xin Eric Wang2   William Yang Wang1
1UC Santa Barbara   2UC Santa Cruz
European Conference on Computer Vision (ECCV) 2022


Abstract

Despite having promising results, style transfer, which requires preparing style images in advance, may result in lack of creativity and accessibility. Following human instruction, on the other hand, is the most natural way to perform artistic style transfer that can significantly improve controllability for visual effect applications. We introduce a new task—language-driven artistic style transfer (LDAST)—to manipulate the style of a content image, guided by a text. We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. CLVA further compares contrastive pairs of content image and style instruction to improve the mutual relativeness. The results from the same content image can preserve consistent content structures. Besides, they should present analogous style patterns from style instructions that contain similar visual semantics. The experiments show that our CLVA is effective and achieves superb transferred results on LDAST.


Visual Attribute Instructions on DTD2 (press Content for all results)

Content Instruction SANet LST ManiGAN CLIPstyler CLVA Style Semi-GT
grayish bluish green smeared paint
zig zag lines embroidered with green and dark
black lines, splashed spotted, water colours
swirly, grey, rounded shapes, blurry, murky
orange red, bright macroscopic, hand-blown bubbles
floating, colorful, white backdrop
light brown
gauzy texture
blonde color, matted, messy, smooth
painting with zigzag orange green pattern
multi-colored polka-dotted fabric, black background

Emotion Effect Instructions on ArtEmis

Content Instruction SANet LST ManiGAN CLIPstyler CLVA Style Semi-GT
i feel chaotic and confused due to the black and gray tones
i've never seen a black and white horse before
the sky is a good contrast to the dark red/brown hill
beside the ocean
the sway of the vibrant trees seems like they are alive
charmed by the beautiful bright
day, at the side of
the pale water
the bright soft
colors reminds
me of a sunset
pen and ink
ornate drawing, something decadent
the gold color
is very energizing
balmy weather, sparkling water, swimmers looking
the yellow boxes
are suspended and very confusing

Specific Content Domain (Car and Church)

Content Instruction ManiGAN StyleCLIP [7] NADA [8] CLIPstyler CLVA Style Semi-GT
piralled, brown, gray, metallic, tunnel
gray rock with
light streaks
pinkish, interlaced, cloth, like
pillow cover
colorful smooth pretty circular round
swirly, silver, rounded shapes, blurry, murky
light green and yellow flannel
plaid fabric
brown salt
deposits forming around crystals
hexagonal, orange, blue, smooth white
soft porous coiled grey spiral
light brown, twisted, type of material

CLVA Results

Instruction
purple pink violet medium polka dots
banded blue
orange lined
ink painting, black dotted line, whiteboard
colorful smooth
pretty circular round
wrinkled, colorful, soft fabric on black background
transparent, white, brown, golden, rocky
sun is shining, bouncing light, wonderful summer
i am confused because of the sunrise in front of overcast sky
i feel chaotic and confused due to the black and gray tones
jazzy and soulful painting, speaks for itself with melancholy undertones
beautiful and vibrant, there is a contrast like the fall colors
dark background with the red figure gives me a feeling of something sinister

High-resolution (2560x1440) Results (right click to view in full size)

Instruction
crystals like a krypton diamond in white and green colors
painted, rubbed, smeared yellow, green, blue and red
wrinkled leather like gray shiny surface
the boats appear lovely in the way they gently rest on top of the peaceful waves
warm painting feels like the sun is setting behind giving it a golden peaceful glow
i feel excited because the shadows suggest the sun is rising and starting a new day

Reference

  1. A Neural Algorithm of Artistic Style. Gatys et al. arXiv:1508.06576.
  2. Universal Style Transfer via Feature Transforms. Li et al. NeurIPS'17.
  3. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. Huang and Belongie. ICCV'17.
  4. Arbitrary Style Transfer with Style-Attentional Networks. Park and Lee. CVPR'19.
  5. Learning Linear Transformations for Fast Image and Video Style Transfer. Li et al. CVPR'19.
  6. ManiGAN: Text-Guided Image Manipulation. Li et al. CVPR'20.
  7. StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery. Patashnik et al. CVPR'21.
  8. StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. Gal et al. arXiv:2108.00946.
  9. CLIPstyler: Image Style Transfer with a Single Text Condition. Kwon and Ye. CVPR'22.