Spellbrush//AI Notes

Check out our brief demo video (0:50, no audio), or scroll further for written notes.

Demonstrated above is a tightly integrated photoshop workflow where human and AI work together to produce the final image. Ruwen is drawing on a Cintiq 27QHD on the right side and cross checking her work with the results of Spellbrush on the left.

Head lines Head raw Head final
Artist Sketch Computer Raw Output Artist Final Touchup (Ship it!)

Working with an AI assistant allows Ruwen to produce final pieces in 5-10x less time.

To give a sense of the speed difference, Ruwen is shown here coloring the original image with no AI-assist. Note that even at the end, the images aren’t 100% comparible as she still has a fair bit of blending and coloration work remaining to do (0:56, no audio).

If the artist is unhappy with the initial colorization, they can force the network to paint the image with a particular palette via a simple layer-based workflow.

Head lines Head hints Head hints output
Original sketch. Color hints pained on new layer. Ship it!

Tech Notes

Dataset is currently trained using synthetic sketch data scraped off the internet. We are also in the process of building a separate novel synthetic dataset based off individual frames of animation from video.

Our implementation influenced by the following papers:

  1. Hensman, P. & Aizawa, K. cGAN-based Manga Colorization Using a Single Training Image. (2017). at http://arxiv.org/abs/1706.06918
  2. Furusawa, C., Hiroshiba, K., Ogaki, K. & Odagiri, Y. Comicolorization: Semi-Automatic Manga Colorization. (2017). at http://arxiv.org/abs/1706.06759
  3. Frans, K. Outline Colorization through Tandem Adversarial Networks. (2017). at http://arxiv.org/abs/1704.08834
  4. Liu, Y., Qin, Z., Luo, Z. & Wang, H. Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks. (2017). at http://arxiv.org/abs/1705.01908
  5. Zhang, L., Ji, Y. & Lin, X. Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN. (2017). at http://arxiv.org/abs/1706.03319
  6. Dong, C., Loy, C. C., He, K. & Tang, X. Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2014).