Check out our brief demo video (0:50, no audio), or scroll further for written notes.
Demonstrated above is a tightly integrated photoshop workflow where human and AI work together to produce the final image. Ruwen is drawing on a Cintiq 27QHD on the right side and cross checking her work with the results of Spellbrush on the left.
|Artist Sketch||Computer Raw Output||Artist Final Touchup (Ship it!)|
Working with an AI assistant allows Ruwen to produce final pieces in 5-10x less time.
To give a sense of the speed difference, Ruwen is shown here coloring the original image with no AI-assist. Note that even at the end, the images aren’t 100% comparible as she still has a fair bit of blending and coloration work remaining to do (0:56, no audio).
If the artist is unhappy with the initial colorization, they can force the network to paint the image with a particular palette via a simple layer-based workflow.
|Original sketch.||Color hints pained on new layer.||Ship it!|
Dataset is currently trained using synthetic sketch data scraped off the internet. We are also in the process of building a separate novel synthetic dataset based off individual frames of animation from video.
Our implementation influenced by the following papers:
- Hensman, P. & Aizawa, K. cGAN-based Manga Colorization Using a Single Training Image. (2017). at http://arxiv.org/abs/1706.06918
- Furusawa, C., Hiroshiba, K., Ogaki, K. & Odagiri, Y. Comicolorization: Semi-Automatic Manga Colorization. (2017). at http://arxiv.org/abs/1706.06759
- Frans, K. Outline Colorization through Tandem Adversarial Networks. (2017). at http://arxiv.org/abs/1704.08834
- Liu, Y., Qin, Z., Luo, Z. & Wang, H. Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks. (2017). at http://arxiv.org/abs/1705.01908
- Zhang, L., Ji, Y. & Lin, X. Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN. (2017). at http://arxiv.org/abs/1706.03319
- Dong, C., Loy, C. C., He, K. & Tang, X. Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2014).