Justin Pinkney

Trailer Faces HQ Dataset

2024-08-17T00:00:00Z

A dataset of 187 thousand high resolution face images from movie trailers! Download it from huggingface. #

Before the advent of giant web scale image datasets FFHQ used to be considered a big dataset full of images of faces on which many, many GANs were trained. One of the many issues with FFHQ (side note: don’t get me wrong there are lots of others, but I was focused on this one at the time) is the lack of diversity in expressions on people’s faces, being scraped from Flickr it tends to mostly be fairly bland smiles all round. At the time I was working on a project related to extracting latent directions corresponding to different emotions and the lack of emotions in ffhq is very noticeable, both in the original dataset as well as the ability of the stylegan model to represent those non smiling/neutral expressions.

Finding emotions #

After this I was on a hunt for a large diverse high resolution dataset of faces with varied emotions, unfortunately none of the exist ones seemed to fit the bill examples?

I guess that meant I had to collect my own. Trying to think of places that had high res closeups of faces in a great variety of emotional states, movies seemed an obvious possibility.

At the time it turned out that the apple trailers website was very easy to scrape, it was literally just a list of links to .mov files, and they were at very high bit rate and resolution, generally showing much less compression than a similar trailer on YouTube. High bit rate is important as it’s not often obvious just how bad any still from even a 4k trailer look with typical web levels of compression. The use of trailers also meant there was a good variety of faces compared to tv shows for example which just have the same few faces over and over again.

TFHQ dataset #

In the end I collected around 186 thousand high resolution face images to make my new dataset, if you’re interested in some of the details they are listed below: ^[1] 1. These bits were expanded from a paper that never went anywhere.

Todo picking which trailers to process

I downloaded all movie trailers and featurettes listed on the Apple Movie Trailers website as of August 2022. That resulted in 15,379 trailers at Full HD (1080p) resolution, amounting to approximately 2 TB/507 hours of video.
to detect faces I used the pre-trained Yolov5-face large model [32] rejecting any detections with bounding box less than 256 px in height, or confidence less than 0.5.
One clear challenge was deduplication, film sequences obviously contain many similar frames and these are often motion blurred due to the low frame rate of movies. That means I would get face detection of the same face over the course of a shot and I needed a way to choose only the sharpest frame and discard the rest. First I computed an image similarity metric using a pre-trained CLIP ViT-B/32 [33] to detect sequences of detections which were very similar. Then I measured the variance of the Laplacian over the images as an approximate relative sharpness metric. The for each set of similar frames I discarded all but the sharpest images, the similarity threshold was tuned by hand, considering frames where there was no significant motion, or significant changes in expression as similar, e.g consecutive frames of a person talking without much head motion should be consider similar.
Finally I did the usual FFHQ alignment on the faces crops [21]. This also involved a lot of frames which needed padding due to the aspect ratio of movies tending to crop the top of the head, for this I did the usual reflection and blur padding used for ffhq.
many of the final images had undesirable properties such as occlusion, non-photographic faces, or overlaid text. To get only the best quality images and remove any false detections from the face detector I did further perform quality filtering by training a classifier on several hundred subjectively determined “good”/“bad” example images, and use this to exclude predicted “bad” images leaving a total of 186,553 images in the dataset.

Face identification #

In order to train my face identity conditioned diffusion model I also wanted to extract face identity information for the dataset and associate different image of the same person with each other for building a retreival based training dataset. I extracted face embedding vectors for every image using a pretrained model from InsightFace_Pytorch (specifically the IR-SE50 model) and then I created a .json file which for every file listed all the other image files with a high cosine similarity of face embeddings of greater than 0.5 (side note I also added a threshold to prevent any matches with a clip similarity greater than 0.9 to avoid very similar images). This gives a dataset where for every image you can easily access the other images of the same identiy in different poses or settings.

This produces a .json file with entries which looks like:

{
  "00000000.jpg": [],
  "00000001.jpg": ["00024932.jpg", "00036845.jpg", ...],
  ... etc

empty lists indicate that a given image has no face id matches. For those with matches the face identifier gives pretty satisifactory results:

Usage #

The dataset is released under the Creative Commons BY-SA license. If you use it for something cool please let me know! And if you want to cite the dataset, here is a bibtex entry:

@misc{pinkney2023tfhq,
      author = {Pinkney, Justin N. M.},
      title = {Trailer Faces HQ dataset},
      year={2023},
      howpublished= {\url{https://www.justinpinkney.com/blog/2024/trailer-faces}}
}

Experiments with Swapping Autoencoder

2024-05-05T00:00:00Z

Back in the GAN days most of my time was spent playing with famous models like StyleGAN and pix2pix. But another of my favourite models, which was much less well known, was Swapping Autoencoder, a lovely model from Adobe research for which all the code and checkpoints were released ^[1] 1. It was also the method (if not the exact model) used for the landscape mixer feature in Photoshop . This is a little post for me to give the model a little of the love it deserves and to record some of the things I made with it.

The basic principle is to try is to generate images but to separate control of the large scale structure from the small scale texture/style details, and to learn this ability without any access to specially created data.

I'm not going to describe how it's trained (the paper is fairly accessible if you're interesting), but in the end the model encodes and image into to parts: a style and a structure code. With style being defined as the idea that multiple small crops should plausibly look like they came from the same image. Then you can generate new images with the model by providing a style and structure code, and the cool bit being that you can arbitrarily swap these between different images to get new unexpected things!

Schematic of the training scheme of Swapping Autoencoder from: "Swapping Autoencoder for Deep Image Manipulation"

The intended way to use the model was to take the content representation from one image, and the style from another to do a sort of "style transfer", although each model was limited to a single domain like landscapes, churches etc. So it was more like a texture/appearance transfer. But there were a lot more interesting things you could do with these broken down latent codes. And I spent a lot of time experimenting witht things and made a tremendous number of images and videos I found fascinating, and here are a bunch of them:

Spatially blending landscapes #

First off it's fun to take two image and encode them into style and content, then squash the two content feature blocks together side by side (with maybe some interpolation where they meet), pick one of the styles or combine them, then generate a new image. You get pleasantly blended impossible landscape images this way.

PCA latent editing #

Another classic technique from the StyleGAN era is to find latent directions in whatever latent space you are working in. GANSpace introduced the idea of encoding a bunch of images then doing PCA to find promising directions. The directions I found encoded in the Swapping Autoencoder latent space are a bit odd, but probably with more effort you could find more semantically meaningful ones. (But who need semantic meaning when you can have weird ones?)

I can't actually remember if this is a latent direction in the style or content space, but either way it's probably roughly the "more clouds" direction.

CLIP guidance #

Around this time was the CLIP guidance craze. I thought I'd have a go using applying the typical CLIP loss to optimise the swapping auto-encoder style latent based on a text prompt. As you can imagine this works pretty well for image editing.

Original image on the left. Optimisation of the Swapping AE style vector with a text prompt of something like "lush green trees" and resulting image on the right.

I also tried optimising the content latent image with CLIP, this turns out weird, but quite interesting. Much less recognisable stuff, but a lot of interesting textures going on.

Unsynchronised style/content video generation #

The autoencoding ability of Swapping AE is sufficiently good that you can auto-encode a video and it still looks fairly good (apart from the texture sticking we are all familiar with from non-StyleGAN3 GANs). I don't know why you would want to do that. But if you offset in time or average or otherwise mess with the style and content encodings relative to each other you can create some interesting video transition effects:

Recursive content transformation videos #

Another fun effect I stumbled across was making videos my recursively encoding and then generating an image over and over, then you can stitch all the generated images together to make a video, like the one above.

It gets more interesting if you apply a transformation to the content encoding between each frame. Something like a simple zoom in/out or a rotation give all sorts of weird and wonderful patterns emerging ^[2] 2. I found that applying a zoom out transform was more effective than zoom in for preserving details and not simply degenerating into noise. Then I reveresed the frames to preserve the zooming in effect. Watch to the end of the second video to see what the starting image was. .

A little schematic of how each video frame is generated with transforms applied to the content encoding ^[3] 3. After drawing this schematic in Procreate I realised I had made it way too low resolution, but Clarity turns out to be a decent upscaler, especially if you use the 'wrong' settings.

.

Train rides #

I also used Swapping Autoencoder for one of my earlier attempts to make synthetic train rides.

https://youtu.be/ITrWUeHqwu4?si=ggp1KoFoimbCmlvt

The end #

Well that's about it, and interesting model with plenty of scope to play with. As with most GANs it's a domain limited model and not general purpose. Maybe in the brave new world of diffusion we could train something that can do anything. But it's not exactly obvious how to train something like this with diffusion, the style training learns to match a distribution indirectly as the loss is defined by the Discriminator enforcing that crops should match the data distribution of other crops. With the direct, incremental loss of diffusion how to do this seems less obvious. Though there are other diffusion based methods which try and seperate style and content. (I'll add a list sometime...)

Image Mixer Diffusion

2024-04-19T00:00:00Z

Just want to play with the model? Try it on Lambda labs or Huggingface

I'm doing more catching up with write ups of past models I trained, this time it's the turn of "Image Mixer" at the time I did at least share the checkpoint and a demo, but here's a more detailed write up of the process of training the model itself.

Even before I worked there I was very impressed with Midjourney's blend feature, where you can combine two (or more) images together ^[1] 1. I'd love to see someone make a version of Infinite Craft but blending images instead of words. . This way of prompting always resonated with me as it gave me vibes of interpolating in StyleGAN's latent space.

In this type of image blending everything gets mixed together in a big soup roughly 50/50, so the style content and objects in the image all end up mixed together between the two images ^[2] 2. Now we have more fine-grained ways of blending images in Midjourney, Style and Character references let you take specific aspects of source images and they are great! . This gives very cool effects but I wanted to explore a more "compositional" method for image blending, i.e. taking the elements from each image and placing them together in a new image. The resulting model was Image Mixer, I don't think it really succeeded, but that's what I was going for, and something fun came out anyway.

To make the model I fine tuned my image variations model, instead of using the clip embed of the whole image to condition the diffusion model, I took several small crops of a single image and used all those image embeds as conditions (concatenating them into a sequence of tokens). The hope being that the model will learn that it has to use the appearance and objects visible in those crops, but will figure out how to compose them together into a reasonable image.

Sometimes it works a bit, but it doesn't quite get there in general. I increased the training resolution to 640x640 to give me space to extract crops which didn't contain too much of the original image. The model does manage to mash different concepts together although the image quality can be somewhat off sometimes, and it still tends to blend the concepts together. The other nice effect is that you can add text conditions as the model is trained to use the shared image text latent space of clip ^[3] 3. Of course you can do most of these things using the base Image Variations model too, and mix images either by averaging or concatenating embeddings .

I also tried training a version which had one global clip embed of the full image and a set of smaller crops. I was hoping that would allow a more style transfer sort of approach with the global embed controlling the content and the local ones more able to affect the details of texture and style. It didn't work out that way unfortunately.

Still it's a fun model to play with, even if it's a bit outdated by modern image generation standards. Take it for a spin, here or here, or download the checkpoint.

Some more examples:

Face Mixer Diffusion

2024-01-19T00:00:00Z

I'm still trying to catch up on writing up some of the "old" diffusion experiments I did and models I made during 2022. This is another quick write up of a model designed to generate new images of a person's face given a single image of that face with no further fine tuning. There are now better models available for this purpose, in particular take a look at IPAdapter face and PhotoMaker.

Mixing faces: taking identity from the left images and 'other stuff' from the top images, make the fake faces in the middle.

Feed forward Dreambooth? #

Shortly after Stable Diffusion was released Dreambooth showed how to fine tune such models for particular styles subjects or people, I've known for a while how much people are obsessed with their own faces, and for a while Dreambooth led to a blossoming of short lived apps and websites for generating images with your own face in (who knows maybe they still exist, people seem to endlessly love their own face).

Dreambooth is somewhat cumbersome as a technique in that it involves multiple example images and training the diffusion model itself. I had fully been expecting someone to come up with a model which could do the same task but without the need for subject specific training, but it seems like that's been a somewhat challenging problem, the first hints of models which can do this (although still not with the level of likeness that Dreambooth could achieve) are only just coming out now.

Of course I had a go myself. The aim being to make a model which you could give a single image of a face and for it to make new images of that person. I took an approach that skipped out the text prompt all together, fine-tuning Stable Diffusion to take an image of a person, crop out the face region, and used a pretrained model to make tha into a vector to use as the condition for generating a new image.

Schematic of the training setup for face conditioned diffusion. This version just extract a single embedding for the face region using CLIP, in some variations I tried pretrained face recognition networks.

I tried a few variations of the concept, not all of which I can remember honestly, but all were trained on datatsets of aligned faces, where I could easily extract a central crop which contained only the face and the rest of the image contained all the "other stuff" (hair, background, accessories, hats, etc.). For all the models I took the central crop and then ran it through an image encoder, either CLIP or a pretrained face recognition network, both would give a vector representation of the face. This can then be passed to the cross attention layers instead of the text embeddings as in the original model. After training like this for a while you can pass a new image of a face, crop and extract the appropriate embedding then use this to generate a new image of (hopefully) the same face.

Results - Making faces #

My first model was trained on FFHQ using an open source face recognition network as the face ID embedding. After training the samples looked promising when I took random faces from FFHQ and asked the network to create new images of the same person.

Left column are all real faces from the FFHQ dataset, and remaining columns are samples generated from the Face ID condition diffusion model given the face ID from the images on the left. These do (sort of) look like new images of the person on the left.

Unfortunately, when I tested in on validation images (faces that the network hadn't seen during training), things weren't so good. The model generated a set of consistant faces of people, but not of the input image. I guess this network had overfit on the original training dataset, FFHQ only has around 70k images, which is tiny by modern standards, I had tried to apply some image augmenation to the faces before passing them to the face recognition network but it seems like this wasn't enough. It gets some features right, there is a clear focus of the ID network on shapes of noses, but these samples don't really resemble the original people at all.

Generated samples using faces from CelebA-HQ datasets, these haven't been seen by the model during training. There are some elements of similarity (if you squint) but the model has clearly failed to generalise.

Back to CLIP #

As an alterative face encoding I went back to trusty old CLIP image embeddings which are clearly very well proved out when it comes to conditioning a diffusion model. I followed the same procedure as above, but used the LAION CLIP-L model to extract the embedding vector correspdonding to the face region. (I might also have resumed from my existing image variations model, but I can't remember).

Generated faces from a diffusion model conditioned on the CLIP image embedding for faces from the FFHQ datset (left).

Generated faces from a diffusion model conditioned on the CLIP image embedding for faces from the CelebA-HQ datset (left).

Trailer Faces HQ and retrieval training #

Around the same time I made a large dataset along the lines of FFHQ, but with data scraped from movie trailers called Trailer Faces HQ (TFHQ) (I'll blog about that sometime too). As well as being a lot bigger than FFHQ, it also had a lot of repeated faces, both from different shots of the same trailer, as well as the same actor in different movies. This meant that if I could find the corresponding sets of images I could train my model with retrieval, in other words instead of just taking a crop of the face and trying to augment it with some image transforms, I could use a different image of the same person.

An example input image on the left, and retreived images of the same actor on the right. I used a pretrained face recognition network to find similar faces, and also use CLIP to avoid including images which are too similar (e.g. from adjacent frames in a video).

Controlling face and background #

Making new images of a person is nice, but at this point I've stripped out all the text control, so the face embedding is the only way to control this model ^[1] 1. I wish I'd done some more experiments like embedding interpolation, SDEdit type image-to-image, or multi-cfg on the different embeddings, but I didn't at the time, and now I've lost the checkpoint! .

To try and add back some control I trained a new version with an extra embedding added corresponding to the background of the image (i.e. all the image with the face region masked out in grey). Apart from that everything is the same, and once trained if you give a single image, and extract a face and background embedding, then you can again generate new images of people's faces:

Schematic of the face conditioned diffusion model. The face region and the whole image with the face masked out are both passed through the CLIP image encoder to produce two embeddings for conditioning the diffusion model.

Real images on the far left, rest are generated. When you use the same image for both the face and background you get more pictures of the same person in a similar setting.

It gets more interesting when you mix the face embedding from one image, with the background embedding from another. In an ideal world you should get an image of the person in image 1, with the hair/lighting/hat/background etc, of image 2.

Mixing faces, face embeddings taken from the images on the left, background embeddings from the images on top, generated mixed faces in the middle.

And it sort of worked! Many of the generated images do appear to be the original person, and you do pick up various features from the the "background" image. Some things change that you might not expect like gender, and other things stay fixed like expression, which is reasonable given the simple way I divided up the image for the different embeddings.

Where did I put that checkpoint again? #

This was all quite a long time ago, back at the end of 2022, but I've only just got round to writing about it all. I was going to share the models too, but unfortunately I seem to have misplaced them, I really have no idea where the saved checkpoints are now, I have a so many hard drives and folders of checkpoints, but none of them seem to be the right ones. Oh well.

If you really want to you could train the things yourselves. All the config files are in my Github repo, and CelebA, FFHQ, and TFHQ are all things you can download.

I also talked a bit about this model, along with a bunch of others, in the Hugging face diffusers talk.

I liked the old StyleGAN Face mixing vibe I got with this model:

https://twitter.com/Buntworthy/status/1592845516127481857

After I tweeted about this blog post a couple of people pointed out some similar work. Bolei Zhou mentioned the great IDinvert from back in the good old days when GANs ruled the image generation roost, and Yong-Hyun Park pointed me the way of this recent interesting paper on content injection using the "h-space" of Unet based diffusion models.

Figure from the paper "In-Domain GAN Inversion for Real Image Editing", amazingly similar to my examples above!

Figure from the paper "Training-free Content Injection using h-space in Diffusion Models". The approach involves some interesting injection of activations into the bottleneck of a pre-trained diffusion model unet.

Würstchen v2 Pokémon

2024-01-07T00:00:00Z

Outputs of the Pokemon fine-tuned Würstchen v2 model for prompts: "Hello Kitty", "Old man looking at the moon", "Donald Trump", adn "Yoda". Top row EMA weights bottom row normal.

The second half of 2023 saw a ton of image generation models come out (including our own Midjourney v6 😊), some open source some not and most interesting in their own way.

One particularly neat little model was Würstchen v2. Würstchen v2 makes the interesting choice of learning a Latent Diffusion Model which is conditional not on text embeddings like Stable Diffusion but on spatial image features extracted by and EfficientNet model, this acts like a very compressed latent representation of the original image. Finally a second diffusion model is trained to generate these spatial image features conditional on text embeddings. Essentially this is training a Stable Diffusion like text to image model but using a super compressed (42x) latent space rather than the typical 8x compression for Latent Diffusion models.

Architecture of the Würstchen family of models, figure from "Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models"

The full pipeline ends up being 3 models: Stage C, a text to very compressed latent image diffusion model; Stage B, a very compressed to less compressed latent image diffusion model; Stage A, the typical latent decoder. With the advantage that Stage B and C can remain fixed for fine tuning, and only Stage C needs to be trained, and given the small spatial dimensions it operates on it can be done so quite efficiently.

To try it out I tested fine tuning with my dataset of cpationed ^[1] 1. My original dataset was captioned using BLIP, which is a quite out of date captioning model by today's standards. If you're after more detailed captions Sayak Paul kindly shared a version captioned with GPT4 Pokemon images (the same one I used to make text-to-pokemon). After a few tweaks to the training script which didn't seem to be set up for thev v2 model at the time (I originally did this around September 2023) ^[2] 2. Since then it looks like there is a training script for the Diffusers version of the model. I haven't used it myself but it's probably an easier place to get started yourself. The example even uses the same dataset but I didn't find any example outputs. , I could run a quick fine-tune and start making Pokemon out of text again. After about 5000 training steps the model behaved in a pretty similar way to my original Stable Diffusion version. You can put in names unrelated to Pokemon and get a "Pokemon-ified" version out.

Here are some more example outputs of "Mario", "Girl with a Pearl Earring", "Boris Johnson", and "Ramen". You can compare some of these to the example outputs in my previous Stable Diffusion based model which overall worked marginally better I think.

Side notes and a Gallery

2023-12-11T00:00:00Z

Sidenotes #

I just added side notes to my blog, thanks to my move to 11ty (from the bloated mess that was Gatsby) and the wonderful sites and articles people have shared on the small web (as well as the power of view source) it's been easy and a pleasure!

I always want to add margin/side/foot notes to my posts, it's probably a sign of a bad excessively detail oriented writing style. Because scrolling up and down is bad, as is hovering, side notes are probably the best solution, the main tricky bit is figuring out what to do on mobile.

Here's some nice sites I looked at for inspiration.

In the end I went for a tap to hide/show the side notes on mobile/narrow screens. And a little visual wiggle on hover on desktop to make clear what the side note number corresponds to.

Gallery page #

I also added a little Gallery page of all the images on my site, given this is a pretty image focussed blog it's nice to have them all in one place.

A helpful comment in a GitHub issue pointed me in the right direction to find all the relevant images in my site and then iterate over them to display in a single page (which you can see here).

I still have a few outstanding things I'd like to improve:

Better layout of the images (hopefully still sticking to plain old css and html if possible)
~~Links back from the images to the blog post where they appear (this is a bit tricky as sometimes I organise the images in subdirectories)~~

UPDATE: With a little help from GitHub copilot it was actually pretty easy to add a link back to the original blog posts from the images. I find them by looking up the file tree for an index.md file that indicates a post, and from that making a templated page I can link to.

Now the css is a bit messy and funky in various cases, but that's the fun of of having a website of your own to play with!

Searching for Generative Train Journeys

2023-11-05T00:00:00Z

I've long had an obsession with trying to create a purely generated train journey with machine learning tools. Cruising through the latent space, inside the internal world of a model is super appealing, and this post documents my years long efforts to do using various generative models along the way.

In the end the rise of diffusion models made what I was trying to do possible, and if you want to see the end result, the thing I'd been striving for all these years here's the final video:

https://youtu.be/_mt8Sbm-7CE?si=VheKzouy54rZeSXN

The StyleGAN era #

As many of the posts on this blog show I spent a lot of time playing with StyleGAN, and some of the very first StyleGAN models I trained myself were trying to generate non-existent train views. Turns out there are quite a lot of high quality YouTube videos of train rides taken from the cab to use as training data. Being the original StyleGAN, and some of the first GAN models I'd ever trained the results weren't great, but they were really exciting to me. Unfortunately they couldn't take me anywhere but the shifting landscapes and scenery in interpolation videos were still mesmerising. ^[0] 0. At the time I tried to fake the forward motion effect with some simple depth estimation and warping, it didn't look terrible but was pretty limited. .

Trying to move forward (pixel2style2pixel) #

Once I had a trained StyleGAN model I wanted to try travelling with it. There are some obvious approaches that I didn't try like trying to find a latent direction of "moving forward", and the method I took was inspired by the fairly convincing appearance of simply zooming into the image with some depth aware warping. If I could take an image generated by the model, warp it slightly to look like we're moving forward, then re-encode that image into the StyleGAN latent space I could recover the "next" image in the imagined journey, and keep repeating the process to create a video. To embed images into the latent space I trained a pixel2style2pixel ^[1] 1. E. Richardson et al., Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation, Jun. 2021. model on my train StyleGAN, then I generated an image, warped it, encoded the warped version in the StyleGAN latent space and generated the new image. Unfortunately my train could never move forward with this method!

Why won't my imaginary train move forward!?

Now I have a bit more experience and intuition of how StyleGAN works, it seems pretty clear that the injected noise maps are what's controlling the positions of small details like the poles and wires visible in the frame. As these noise maps are fixed during my video sequence, nothing really seems to change or move forward.

Swapping autoencoder #

After that failure I gave up on StyleGAN ^[2] 2. I did try one further StyleGAN based method which was generating infinite scrolling videos using Aligning Latent and Image Spaces to Connect the Unconnectable (ALIS), unfortunately without the effective of parallax the videos don't give the effect I wanted. , and instead of trying to generate a fully synthetic video, realised I could use a different model to fake the effect with a real video as a starting point.

A highly under-rated model of the GAN era was Swapping Autoencoder, it was trained to distentangle style from content so you could take two images and take the content/layout from one and apply the style of another. Like most GANs it worked best under a limited domain, and the released landscapes model was amazing ^[3] 3. T. Park et al., Swapping Autoencoder for Deep Image Manipulation,” 2020. .

Examples of Swapping Autoencoder outputs from the paper: "Swapping Autoencoder for Deep Image Manipulation".

I realised I could fake the effect I wanted by taking a real life video of the view from a train ride and applying the style of random other landscape photos, melting between them as the train traveled. It didn't quite achieve my goal of traveling real imagined landscapes but the effect was great.

https://youtu.be/ITrWUeHqwu4?si=ggp1KoFoimbCmlvt

Next frame prediction with Stable Diffusion #

Once stable diffusion came out and I had played with fine-tuning it a little I came back to my usual quest. This time I was took some side facing train video of my own and tried training a next frame prediction model. If you're not familiar with Next Frame Prediction (NFP) then it's essentially training a network to take in one image, and to predict another image that would be the next frame in a video. You can see a good tutorial on how to train your own by Derrick Schultz. The use of NFP models for making train video is very old (by machine learning standards), as seen in this amazing (and very inspirational) work by Damien Henry.

A schematic of the NFP version of Stable Diffusion.

After a while of training to predict the next frame from my tiny train dataset the results were almost too good. Recursive next frame prediction to make a video actually turns out so realistic it's kind of boring.

https://youtu.be/qGqPJSCuxPw?si=UwFWlDmcMCKQa0vu

This is a video is made by giving the NFP Stable Diffusion model a single starting frame example, then continuously getting it to predict a new frame, and then feeding that frame back in to create a video.

To make it more interesting I needed to add some more levers of control. To do this I repurposed my image variations version of stable diffusion, and again fine-tuned it on the next frame prediction task. But this time instead of a only receiving the previous frame as input it also gets the clip image embedding of that frame. At inference time this lets you change the image embedding to anything you want which controls much of the style of the image, whereas the previous frame is most important for the global structure. This gives dream like videos of traveling sideways but with shifting and changing styles, colours and content depending on the clip embeddings used at each frame.

https://youtu.be/8VLWoO232w8?si=PAEqlOkvCVhuhgVp

The final cut #

Putting all the pieces together I generated long sequences of frames where I would change the clip embeddings to that if various images and artwork. Sifted through for the section that worked best together and put them all together with an soundtrack assembled from various free audio samples of train noises.

You can jump to the final here, otherwise there are also some nice still frames that come out of this technique:

Stable Diffusion Image Variations

2023-08-27T00:00:00Z

This post is time-travelling a little. For some reason I never blogged about the image variations stable diffusion model I trained, so this is a bit of a recap on the how and why of the image variation model and to collect some links and experiments using it. The model itself is a little old now and there are other models which are similar might be better like Karlo v1.0 and Kandinsky (probably not the Stable Unclip though ^[0] 0. For some reason the Stable Unclip is actually terrible quality. This is an example of the output when using my typical Ghibli house image for testing (hint it's really bad) .). But on my drive to try and own my content and make sure it exists beyond walled gardens like Twitter, here's a post mostly cobbled together from other sources.

As soon as the original stable diffusion was released I wanted to see how I could tweak the model beyond regular fine tuning. The image variations shown in the original dalle2 paper were always really compelling and this model presented the first chance to actually reproduce those (without lots of training resources). The key part would be to somehow pass images encoded as clip embeddings to the model rather than text embeddings.

Schematic of how the normal Stable Diffusion uses a set of token embeddings from the CLIP text encoder to condition the diffusion model.

Originally I mis-understood the architecture of stable diffusion and assumed that it took the final clip shared latent space text embedding and thought I might just be able to swap this out for an image embedding, but actually stable diffusion takes the full sequence of pooler and token embeddings, so I couldn't simply swap them out ^[1] 1. I did do some experiments trying to generate these word embeddings from image embeddings but they never panned out .

So instead I decided to swap out the conditioning altogether and fine tune the model to accept projected image embeddings from clip. So instead of using the text encoder to make a set of (batch_size,77,768) dimension CLIP word embeddings, I used the image encoder (plus final projection) to produce a (batch_size,1,768) size image embedding. Because this is just a shorter sequence it's easy to plumb into the existing cross-attention layers.

Schematic of Image Variation model showing the replacement of the CLIP text encoder with the CLIP image encoder, conditioning on a single image embedding.

With some simple tweaking of the original training repo I could finetune the model to accept the new conditioning, there are more details on the training in the model card, but here's a quick summary ^[2] 2. This is actually the training procedure of the v2 model which was trained for longer and more carefully, giving better results :

Fine-tuned from the Stable Diffusion v1-4 checkpoint
Trained on LAION improved aesthetics 6plus.
Trained on 8 x A100-40GB GPUs
Stage 1 - Fine tune only CrossAttention layer weights
- Steps: 46,000
- Batch: batch size=4, GPUs=8, Gradient Accumulations=4. Total batch size=128
- Learning rate: warmup to 1e-5 for 10,000 steps and then kept constant, AdamW optimiser
Stage 2 - Fine tune the whole Unet
- Steps: 50,000
- Batch: batch size=4, GPUs=8, Gradient Accumulations=5. Total batch size=160
- Learning rate: warmup to 1e-5 for 5,000 steps and then kept constant, AdamW optimiser

Results #

The very first results I saw during after letting run for a decent amount of time looked like this:

The top row is the conditioning images, and the bottom row are generations from the model. They look "similar", that means it was working!

The first batch of results I shared on Twitter showed the classic image variations for some famous images (see if you can guess which ones):

Version 2 #

After retraining the model a little more carefully ^[3] 3. For v2 I fine tuned only the cross attention weights initially to try and help the model to better adapt to the new conditioning without degrading its performance, then followed that up with a full fine tune at a large batch size. Maybe if I were doing it now I might trying using LoRA. , for longer, and with a bigger batch size, I released a V2 model which gave substantially better results.

Here are some side by side examples of the improved quality of the v2 model, showing the improved fidelity and coherence of the v2 images.

Comparison of v2 (left) against v1 (right) conditioned on various images, one per row ^[4] 4. Original images: .

Quirks (a.k.a the problem is always image resizing) #

One thing that really baffled me for a long time, was that in the huggingface diffusers port I would get much worse quality images which were always slightly blurred. After scouring the code for what might be the problem it turned out to be the same problem it always is in image processing code, the resizing method. Seriously, if you have an image processing issue, check your image resizing first!

Turns out I accidentally trained the model with antialias=False during resize. So when the huggingface diffusers pipeline applied the "proper" behaviour, i.e. using anti-aliasing, the results were terrible, seems like the model is very sensitive to the resize method used.

Running the model with the resize method it was trained on (first) vs a slightly different one (second) gives a dramatic difference to the results ^[5] 5. The input image for this is my typical Ghibli House test: .

The other thing I noticed is that my nice example at the top of this post (Girl with a Pearl Earing by Vermeer) didn't produce pleasant variations as before, but now was badly overfit. It's actually a sign of how badly undertrained the first model was, as that particular image occurs many many times in the training data (see these search results).

Follow ups #

Hopefully the model has been somewhat useful for people, it also showed how adaptable Stable Diffusion is to changing the conditioning, something I've done a bunch of further experiments on. Some of these I've already blogged about: doing CLIP latent space editing and various other experiments.

It was a very popular Huggingface Space, and if you want to try the model yourself, it's probably the easiest way.

It also made an appearance in the paper Versatile Diffusion: Text, Images and Variations All in One Diffusion Model (arxiv). Unfortunately they had done the work before I'd released the v2 model.

And the concept was also the basis for my Image Mixer model (which I'll write up in more detail in future).

It's also possible to use this model as a full on text to image generator by using an existing CLIP text embedding to image embedding prior. I did a little experiment with the LAION prior to show this was possible, but since then the Karlo v1.0 and Kandinsky priors have come out which are probably much better.

Finally if you want to watch a whole talk my me on this and related topics, I gave one at the Hugging Face Diffusers event:

https://youtu.be/mpMGwQa7J1w?si=ilfxsLlQlquIZmB9

The Other Web

2023-08-22T00:00:00Z

The other web #

The recently accelerated enshitification of Twitter made me realise that too much of what I do and make is trapped in that platform. I feel this especially when I start trying to explain projects or experiments I've worked on to people by sharing tweets, why isn't it on my blog?

Its probably not an understatement to say that Twitter has changed the course of my career, it's how Toonify went viral and me getting jobs at Lambda and Midjourney both started with Twitter DMs.

But with recent changes to Twitter making me think a bit more about why and what I'm sharing on the web I increasingly realise the internet doesn't just have to be big platforms, social networks, and the first page of a Google search. There is another web out there.

The other web #

I don't know what the right term for it is, but I've heard various ones:

the slow/personal/handmade/outer/small/indie web
digital homesteading/gardening (well maybe "digital gardens" has been taken by the person productivity/tools for thought crowd)

There's a lot I like about the ethos of tending a little personal place on the web and filling it with things that interest you. Hopefully a few people stumble across it and have that same great feeling of discovery that I do when I come across a lovely home made website full of curiosities and personality.

So I'm trying to share more of what I do here rather than Twitter, and preserve things that were only throw away threads as blog posts (starting here), and make my website my home on the web.

Links #

We need links, more links, to good things not the usual garbage listicles, click bait and ad vehicles. The authentic, real, handmade web is hard to find unless we all link to it. So here's some of the interesting places/articles I found delving into and around this topic:

mmm.page and Kinopio and the communities around them are two of the places that have a wonderful ethos around what web software (beyond pure webpages) can be and helped me find lots more interesting threads
html.energy has a fantastic podcast with interviews with many interesting people
My website is a shifting house next to a river of knowledge. What could yours be? Poses great questions about what an analogy for a website might be, "Website as a puddle" anyone? (from the great Creative Independant)
Handmade Web reading list
An essay on the Small web
32-bit cafe: "a community of like-minded website hobbyists and professionals helping to make the personal web fruitful and bountiful again, full of self-expression and removing the capitalistic drive out of it"
Darius Kazemi's Twitter archive An example of, and tool for breaking your archive out of Twitter
Rediscovering Discovery on the Web
A well curated old fashioned blogroll
Some great examples of public collated rss feeds from blog rolls:
An article on the "small web"
Pages with many great links:

Algorithmic Film Making

2023-08-13T00:00:00Z

https://youtu.be/rsxUcmHNH9s

I recently signed up to Derrick Schultz's online algorithmic film making class. I'm excited to be taking one of Derrick's classes, I've been a big fan for a long time of both his work and teaching (not just because his YouTube channel has long been my default response when people would ask me "How do I StyleGAN?").

I'm also keen to have something I can focus on as an outlet for some ml based art and maybe some open source models and tools. The slow death of Twitter and starting at Midjourney has meant I've not had much motivation or reason to play in the open source creative ml space for a little while now. I also have a 2 TB dataset of movie trailers sitting on my hard drive I want to knead into something interesting.

I don't have any particular ideas for what my final project for the course will be yet, but I've always found the idea of visual collage and montaging videos particularly interesting (as well as the extreme extrapolation of that process: slit scanning).

This is an example of the sort of cut-up montage video that I've been thinking about:

https://twitter.com/Buntworthy/status/1459838418339471364

This originally sprouted from some earlier experiments in collaging images. Here's a link to a thread of some of my experiments and examples from others that convey the sort of aesthetic I like. (One day I'll break them all out from the prison of Twitter into my blog).

https://twitter.com/Buntworthy/status/1443314544252686338

Why this? I'm not, sure something about how it relates to what I do every day: machine learning is all about datasets and seeing the patterns in those. That's what those montages feel like, trying (but not really being able) to convey a sense of the whole as much as any part. (I also spend a lot of my days looking at great big grids of beautiful images. So my eyes end up tuned to the things)

The logical extreme of cutting up images or video and recombining them are also visible in some of the slit scanning and colour sorting experiments I've done before.

https://youtu.be/zgxHXQaObFg

Given that I used to everything related to StyleGAN, here's a StyleGAN related example, slit scanning through interpolation spaces:

https://twitter.com/Buntworthy/status/1451105403857739776

(There was also an interactive version of this I never published:)

https://twitter.com/Buntworthy/status/1222945471003602944

Week 1 - Randomising clips #

The first week's exercise is to chop up something into shots and randomise the order, just to get a feel with working with video datasets and to see what interesting things turns up. I didn't want to face battling with my big dataset just yet, so I cut up The Tale of Princess Kaguya into shots and arranged them in a grid. The grid view fits my aesthetic sensibilities and also means that I can actually watch the entire resulting clip in a reasonable amount of time.

I made two versions, one with a totally random order, and the other where the clips are sorted by order of occurence, but the new clips are placed in the grid as soon as any previous one finishes, so the whole thing ends up going slowly out of sync as time progresses, which I think is a nice effect. The sorted version is at the top of this page and the randomised one below:

https://youtu.be/Lyg16nGbUtM

Flowers Deconstructed

2023-07-10T00:00:00Z

Classic paintings of flowers cut and re-arranged.

A long time ago I made these digitally cutout and collaged images of flowers. They were all created using public domain flower paintings, U2net for segmentation and image magick for collaging.

Experiments in Image Variation

2022-12-17T00:00:00Z

I've been doing a bunch of quick experiments for my improved CLIP Image conditioned version of Stable Diffusion v1. We can actually do lots of fun stuff with this model including transforming this Ghibli house into a lighthouse!

First off you can find the actual model on huggingface hub. Throughout the post we'll be using this crop from a Ghibli frame as our test image.

This is what the standard variations model produces when you feed this image in:

That was classifier free guidance scale 4, we can play with different levels of guidance: (0, 1, 2, 4, 8)

The image is turned into a CLIP image embedding, this is the condition vector. One thing I noticed is that because I set the unconditional embedding to zeros, scaling a condition vector has the effect of increasing its "strength", this effect is different to the usual classifier free guidance. We can play with the length of this vector to control the "strength" of the conditioning by multiplying the vector by 0.25, 0.5, 1, 1.5, 2:

We can also mix different embeddings by averaging them together. Let's mix between the embedding for our Ghibli image and a matisse painting. 0%, 25%, 50%, 75%, and 100% matisse:

Another way to mix is to concatenate the embeddings, yes we can have more than one if we want to. Here we do the following

ghibli, ghibli
ghibli, uncond
ghibli, matisse

We can also use text embeddings to mix with our image embedding, we can mix with the word "flowers" to add flowers. (The variations model actually works fine with text embeddings btw). At the far right it's only using the text "flowers" (we re-invented a text to image model!). 0%, 25%, 50%, 75%, and 100% "flowers":

We can do the same trick of concatenating the embeddings instead of averaging, it's a bit funny though

ghibli, flowers
ghibli, flowers, flowers
ghibli, ghibli, flowers
ghibli, ghibli, flowers, uncond, uncond

More natural is to maybe use a direction in text embedding space to edit our ghibli conditioning. We can use the direction "still water" -> "wild flowers" and add this at different scales to your original conditioning. Here it's adding 0%, 10%, 25%, 50% and 100% of the edit vector "still water" -> "wild flowers":

Changing tack we can experiment with adding noise to our condition vector with scales = 0, 0.1, 0.2, 0.5

Or use multiplicative noise instead, scales = 0, 0.2, 0.5, 0.75

If we take random crops when computing the embeddings we get zoomed in images mostly, a bit boring.

Image inversion #

Now we play with DDIM inversion, followed by editing, as shown in the DALLE-2 paper. We invert our image using standard DDIM inversion as in the original stable diffusion repo., we need to use lots of timesteps. Then check we can decode it to pretty much the same image (if we use cfg it's more saturated). To get good inversion results we use: ddim_steps = 500, start_step=1, cfg scale = 3 (careful with cfg scale! not too high)

For decoding you don't need to use so many timesteps, so it wont take so long, decode with 50 timesteps and you get the original picture back

Now we use our same text diff as before to replace the water with flowers! 10%, 20% 50% 100% and 150% edit:

Or how about turn the house into a lighthouse? 10%, 20% 50% 100% and 150% edit:

We can also use ddim_eta to make variations of our original, but ones that closely match in composition. Here each row is for ddim_eta 0.3, 0.6, and finally 1:

We can make more of these steadily ramping up the value of eta and make a variations video:

Last of all we can combine subtle variation (ddim_eta=0.4) with text diff editing: "ghibli matte painting" -> "dslr leica photo"

Latent editing with image variations

2022-10-31T00:00:00Z

Playing with the variations model a bit more. You can take the CLIP image embedding of an image then do latent editing with a direction in CLIP text space.

original
variation
"colour oil painting" - "bw photo"
"modern dslr photo" - "old bw photo"

original
"old woman" -"young woman"
"modern dslr photo" - "oil painting"

original
"detailed painting" - "child's drawing"
"archviz render" - "child's drawing"
"castle" - "house"

converted from a twitter thread: Mon Oct 31 20:33:59 2022

Text to Pokemon Generator

2022-09-08T00:00:00Z

If you want to just try out the model look here:

All of this work was done as part of my role at Lambda Labs and all the real details of how the model was made, and how you can make one yourself are in this post on the Lambda Examples repo. This blog post just has a few extra details and notes on my experience

Girl with a pearl earring, Cute Obama creature, Donald Trump, Boris Johnson, Totoro, Hello Kitty

There's been much excitement about Stable Diffusion's recent release as you might expect. Sure it's great at making some interesting images but there are a lot of possibilities than just putting in words and getting out pictures. To me it feels like a similar moment to shortly after StyleGAN was released: A powerful, high quality model, publicly accessible, and simple enough to for people to play with, adapt and fine tune for their own purposes.

I've been doing some experiments with the model since it came out. One was changing the input conditioning to enable "image variations" with Stable Diffusion, but this post is just a little summary of some more straightforward initial experiments in simple fine-tuning of the model to turn everyone into a Pokemon!

Catastrophic forgetting? #

One of the amazing things about this model is that it manages to remember some of the "general knowledge" of the original stable diffusion. It doesn't seem obvious that it should do so after training for a few thousand steps on such a limited dataset, but there's a trick involved here! When fine tuning on the Pokemon the model actually starts to overfit quite quickly, and if you sample from it in a naive way it just produces Pokemon-ish gibberish for novel prompts (it has catastrophically forgotten the original data it was trained on), but like many modern networks Stable Diffusion keeps an exponential moving average (EMA) version of the model during training, which is usually used for inference as it gives better quality. So if we use the EMA weights we're actually using an average of the original model and the fine-tuned one. This turns out to be essential in order to turn all those famous people into Pokemon. You can even fine tune this effect by directly averaging the new model with the weights of the original to control the amount of Pokemonification.

https://www.twitter.com/Buntworthy/status/1567804278949007360

Turns out that this is basically the same mechanism that enabled toonify yourself, fine tuning and averaging model means you can end up with an effective mix of the original content with the style you fine-tuned on!

More experiments #

It does seems a bit excessive to fine tune the whole unet as I did in the example above. There are probably lots of different strategies you could adopt trying to freeze different parts of the model, or even try something like layer swapping. One little experiment I tried along these lines to to fine tune only the Attention layers in the the unet. This version helps to preserve the original abilities of the model, but reduces the quality of the Pokemon produced. It's neat the model still remembers how to produce non-pokemon images, but this isn't quite what we want for this application, as it's good that the model ends up producing nothing but Pokemon!

Left is the fully fine-tuned model. Right is attention layers only. The right model can clearly generate a more "normal" yoda, but is less good at making Pokemon.

An obvious next step is to try and compare the above fine-tuning with a method like DreamBooth. If anyone gets round to trying this before I do, please let me know how it goes!

Vivillon is the Rick Astley of Pokemon #

Some people pointed out that the model has a tendency to randomly produce a certain Pokemon, Vivillon for seemingly random prompts. Even when it doesn't make Vivillon exactly it still has a strong preference to make circular radially patterned Pokemon when the prompt is a bit more abstract.

https://www.twitter.com/JanelleCShane/status/1575855505922088960

Someone pointed out that Vivillon actually has several variations, each of which is a different Pokemon, this means it appears a bunch of times in the original dataset. The model clearly ended up overfitting to Vivillon, I might make a version where I remove all but one Vivillon, but right now I quite enjoy that Vivillon occasionally appears when you least expect it like a Pokemon Rick Roll.

https://www.twitter.com/MrCheeze_/status/1575857534874705920

Coverage #

I tweeted about the model when I released it, but seems like it took a little while before it really took off. In particular this tweet seemed to really reach the right audience:

https://www.twitter.com/JDune5/status/1574143254366388232

Since then it's been featured here:

and plenty of other places.

Style Space Face Editing

2021-04-08T00:00:00Z

Repository: https://github.com/justinpinkney/pixel2style2pixel/

A little while back I ported a couple of recent machine learning models to Runway: "Encoding in Style" (aka Pixel2Style2Pixel or PSP)[^psp] and "Style Space Analysis"[^style-space]. They work well together, PSP for encoding an existing image into the latent space of a Generative model, and Style Space Analysis for editing that image in the latent space of StyleGAN. Here's a post on some of the technical details of how it works.

Pixel2Style2Pixel #

PSP demonstrates a highly effective method for encoding real images in the latent space of a pre-trained generative model. The repository provides a pre-trained encoder for finding the latent vector corresponding to an image your supply. There is also some cool follow up work called Encoder4Editing which might be even better for this application.

They even use the good old Toonify model in their research as part of demonstrating the flexibility and generality of their face encoding approach.

Style Space Analysis #

StyleSpace Analysis is a paper with a nice idea which seems to not have gained much attention. It shows that as well as the typical Z and W latent spaces of StyleGAN, there is another latent space: StyleSpace, which provides extremely localisable edits to images. ~~Unfortunately there is no code accompanying Style Space Analysis, but~~ There is now code for StyleSpace analysis! The channels used to edit various attributes are provided in the paper and delving into the PyTorch version of StyleGAN 2 using forward hooks to modify the appropriate values is pretty straightforward.

Other code #

I also use some existing code for face alignment from the FFHQ-dataset repository to ensure faces are aligned as expected by the StyleGAN model. I combine this with a pre-trained face detector from dlib, and add some extra code to "undo" the alignment to replace the edited face back in the original image. This involves some fairly straightforward code using OpenCV to estimate the inverse transform and apply this to align the face back into the reference frame of the original image.

Let's edit! #

Now with these various bits of model code put together and stuck into the nice interactive RunwayML interface, it's easy to do some Face editing. PSP provides the means of embedding a set of real world faces in the latent space of StyleGAN2 and StyleSpace analysis lets us modify some interesting attributes. By placing this edited face back in the original image we can create some fairly realistically edited images.

Here are some examples of facial hair, expression, and make up editing.

The encoder is also flexible enough that it can work with some non-photographic images too.

As the output of the model is constrained by the pre-trained generative model, it can also take unrealistic faces and translate them into realistic ones. It can then be used for blending composite "identikit" type faces, or even translating crude overlays (such as drawn glasses or cartoon moustaches) into realistic images.

This is actually the same technique I used to mock up a face editor app integrated with GIMP

https://vimeo.com/536038892

There's also no particular reason why this should be limited to faces. I'd love to see more of these techniques applied to editing landscape images, but this would require a high quality landscape model trained using StyleGAN2...

Could try harder #

This is pretty quick and dirty, but effective. There are some pretty obvious things that could make it nicer

Blending the composite image #

After compositing the image back into the original, the borders are generally fairly obvious. A straightforward method to improve this would simply be to perform poisson blending of the modified and original image. Or a more sophisticated method could involve segmentation of the person or the attribute region of interest from the modified image to remove the background before compositing back into the original.

Poor embedding #

PSP hates bald people. It believes they should all have comb-overs, check out the image of Vin Diesel (encoded with no edits) below. A bit of extra optimisation of the encoded latents based on a perceptual similarity measure of the original image (e.g. LPIPS) might re-find the baldness.

In other cases I'm not sure any further fine tuning would help. I'm pretty sure the FFHQ model doesn't know anything about tongues.

Batching #

The current code just processes each face in the image individually. It would be a bit more efficient to batch these up. However I don't know the configuration of the remote GPU machine Runway uses, so don't know what batch size I could work with without risking an out of memory error.

References #

[^psp]: Richardson, Elad, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. 2020. “Encoding in Style: A StyleGAN Encoder for Image-to-Image Translation.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/2008.00951.

[^style-space]: Wu, Zongze, Dani Lischinski, and Eli Shechtman. 2020. “StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/2011.12799.

Ukiyo-e faces dataset

2020-10-29T00:00:00Z

Download the dataset: V2

As part of my paper Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains[^rdgi] I use a dataset of Ukiyo-e face images for training a StyleGAN model, this post contains a link to, and details of that dataset.

Updates #

V2 - Removed 28 bad quality images (poor alignment or not face).
V1 - Initial release used in the paper Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains

The dataset #

The ukiyo-e faces dataset comprises of 5209 images of faces from ukiyo-e prints. The images are 1024x1024 pixels in jpeg format and have been aligned using the procedure used for the FFHQ dataset. Above is a map of (almost) all the images in the dataset, images are plotted such that similar faces appear close together[^map]. The images have been downscaled to 256x256 for display.

Further details #

Images are scraped from several museum websites, I then used Amazon Rekognition to attempt to detect faces and facial landmarks from each image. Rekognition does a reasonable job at both tasks, but is clearly imperfect, many faces are missed and there are alignment errors in many of the images. Many of the images are not of very high resolution, so to produce a useable dataset at 1024x1024 resolution I use a pre-trained ESRGAN[^esrgan] model trained on the Manga109 dataset to upscale the images where required, these leaves some artifacts but generally does a good job.

Other datasets #

KaoKore is another dataset of Ukiyo-e faces[^kao], it is more varied and labelled, however the image resolution is lower and faces are not aligned.

License and usage #

This dataset is provided under a Creative Commons Attribution-ShareAlike 4.0 International License.

If using the dataset please cite as "Aligned ukiyo-e faces dataset, Justin Pinkney 2020" or for a bibtex entry:

@misc{pinkney2020ukiyoe,
      author = {Pinkney, Justin N. M.},
      title = {Aligned Ukiyo-e faces dataset},
      year={2020},
      howpublished= {\url{https://www.justinpinkney.com/blog/2020/ukiyoe-dataset}}
}

[^rdgi]: Pinkney, Justin N. M., and Doron Adler. ‘Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains’. ArXiv:2010.05334 [Cs, Eess], 20 October 2020. http://arxiv.org/abs/2010.05334.

[^map]: To generate this image I first extract CNN features from each image using a ResNet50 pre-trained on Imagenet. These high-dimensional feature vectors are then projected into two dimensions using UMAP, grid assigment is then done using the lapjv algorithm.

[^esrgan]: Wang, Xintao, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. ‘ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks’. ArXiv:1809.00219 [Cs], 1 September 2018. http://arxiv.org/abs/1809.00219.

[^kao]: Tian, Yingtao, Chikahiko Suzuki, Tarin Clanuwat, Mikel Bober-Irizar, Alex Lamb, and Asanobu Kitamoto. ‘KaoKore: A Pre-Modern Japanese Art Facial Expression Dataset’. ArXiv:2002.08595 [Cs, Stat], 20 February 2020. http://arxiv.org/abs/2002.08595.

Making Toonify Yourself

2020-09-20T00:00:00Z

If you'd like to keep Toonify Yourself free for everyone to play with, please consider donating to cover running costs at Ko-fi:

So Doron Adler and I recently released our toonification translation model at our Toonify Yourself website. It turned out to be pretty popular with tens of thousands people visiting in the 22 hours it was running for, submitting almost a quarter of a million images for toonification.

https://www.twitter.com/Buntworthy/status/1306236896125870080

regex to capture the above twitter link is:

It got quite a bit of interest on social media, and picked up on a few websites. Unfortunately we had to turn off the toonification server before costs started to get out of hand, but we're working on bringing it back so people can carry on playing with the model for free.

A lot of people have expressed interest in how the model work and how the website was run. So here's a blog post with some details on the traffic and running costs, as well as the technical details of how to run a deep neural network in the cloud serving tens of thousands of requests an hour!

Making an efficient Toonification model #

If you want to know about the details of the original Toonification model, see this blog post.

The original Toonification method involved an expensive optimisation process to encode a person's face using the blended StyleGAN model which can take several minutes to run even on a GPU. Clearly this wasn't going to cut it as a web app! A common pattern in deep learning is replacing expensive optimisations with more neural networks[^style-transfer]. We used the basic idea described in StyleGAN2 Distillation for Feed-Forward Image Manipulation[^distillation], i.e. training a pix2pixHD model to apply the transformation to any arbitrary image, rather than first having to perform the optimisation step.

The novel part here is that the pairs of images we use for the training process are pairs produced by the original FFHQ model and the blended model[^new-model]. Although the pix2pixHD model is only trained on image generated by the two StyleGAN models, when it's done we should be able to apply it to any image and get the same toonification result

Deploying the model #

So after the initial interest on Twitter about my experiments putting together a local web app for myself to run the Toonify model I decided to take a crack at putting up a website where anybody could run it on their own images.

First things first I wasn't going to be running anything on a GPU, so I needed to get the pix2pixHD model runnable on a CPU. The original pix2pixHD repo has some bugs which prevent inference without a GPU and I fixed these on my fork if anyone is interested. In the end I actually decided to export the model to ONNX format so I could run it using the ONNX runtime. This makes the dependencies more lightweight than trying to run using PyTorch and (I hope) the ONNX runtime is built with performance in mind.

I went for Google Cloud Run as a means of deploying the web app. All the thing needs to do is accept an image, run inference, and return the result. It's totally stateless so a good fit for the Cloud Run model. The use of Docker containers in Cloud Run meant that it was easy for me to bundle up any required dependencies, scalability was built right in and there is a generous free allowance (not generous enough it would turn out!).

So after a few evenings of putting together a small app using Flask and Bootstrap, things were ready to deploy!

Toonification in the Wild #

So after some beta testing with friends I announced the release of the Toonify Yourself website on Twitter. It quickly got some reasonable traffic and people seemed to be enjoying trying the model out on themselves.

Some were complaining that their faces were never detected no matter what they submitted, and I fairly quickly figured out (and many helpful people online started to point out) that it was an issue with image rotation on iPhones[^transpose].

By the next morning traffic started to really pick up, partly due to getting on the front page of Hacker News. I was starting to get a little bit twitchy seeing the number of containers spun up on Cloud Run steadily increasing. As lunch time approached we were getting close to 25,000 page views an hour, at times this was requiring 100 containers to service the traffic, and thing were going up fast.

The measly number of free cpu and ram minutes had long since evaporated, and I was getting a little concerned about what the cloud bill was going to be after I came back from an afternoon out of the house. So rather than limit things to a level that most people would get a non-response from the site, I decided to turn off the model and switch to an apology message.

The numbers #

In the end we had close to a quarter of a million page views before I shut down the toonification. Not every one of these corresponds to someone submitting an image, but it's not far off, and each user submitted around 3 or 4 images for toonification.

So how much did this all cost? Not quite as much as I thought, I had set a personal limit of spending around $100 on this and I didn't break that. But here are some details of what it costs to run a service like this.

The model takes about 5 seconds to run inference on whatever hardware Cloud Run uses, and occupies around 1.4 GB of memory whilst doing it. It also takes a slightly astonishing 20 seconds to load the model the first time a container is brought up (and this happens more than I'd like), and the memory peaks at well over 2GB during this period. All this meant that processing a thousand images probably costs around 30 cents (footnote: there are also some other smaller costs like network egress to think about, but that's not much extra), which isn't too bad, but when you're trying to process 25,000 an hour, starts to add up fast!

I'm still pretty amazed that it was so easy to build a site which could service so much traffic and do some serious image processing in the cloud. I've never used any of this scalable serverless technology before, but it was incredibly easy to get going!

Feedback #

A lot of people commented that the images produced didn't preserve enough of the original character of the person, that they ended up looking pretty generic, and that a human cartoonist could do a far better job. I fully agree, there is not way deep learning is going to outperform a skilled artist any time soon! But there is also no way you could get skilled human artists to make cartoon versions of people for 30 cents per thousand.

Despite me putting a line in the FAQ assuring people I was not storing or collecting their images (how on earth would I have afforded that!?) several people commented on it with scepticism, surely something free online must be harvesting your data somehow? But honestly once the toonification was done the original image and the result were gone forever (for me at least). Plus I don't really see why people were worried, people have already happily uploaded millions of photos of themselves to the internet and social media sites, who explicitly do make money from your data. If companies want to collect face data, a silly website like mine is not the way to do it!

The future #

UPDATE: Toonify Yourself is back thanks to the fantastic support of generous supporters on Ko-fi, as well as model hosting thanks to DeepAI. There's still costs involved in running the site, so if you'd like to help keep things free for everyone to play with please think about supporting me on Ko-fi.

Twitter #

Lots of people shared fun examples on Twitter, here are a few:

Coverage #

Here are some links to the coverage this got.

and many many more...

I was even interviewed on the excellent Cold Fusion YouTube channel:

https://youtu.be/KZ7BnJb30Cc

[^style-transfer]: For a classic example see the work on neural Style Transfer.

[^distillation]: Viazovetskyi, Yuri, Vladimir Ivashkin, and Evgeny Kashin. ‘StyleGAN2 Distillation for Feed-Forward Image Manipulation’. ArXiv:2003.03581 [Cs], 7 March 2020. http://arxiv.org/abs/2003.03581.

[^new-model]: Doron actually spent some time assembling new datasets and training a new models so the results you see are a bit different to the ones I originally shared.

[^transpose]: Portrait images are actually saved as landscape on iPhone and the rotation is embedded in the metadata. You can apply this rotation in Pillow using ImageOps.exif_transpose

Toonify yourself

2020-09-01T00:00:00Z

TLDR: If you want a Colab Notebook to toonify yourself click here:

If you're interested in how the website Toonify Yourself works, see this followup post

In a previous post I introduced the idea of Layer Swapping (or more generally network blending) for StyleGAN models. I briefly pointed to a fantastic model created by Doron Adler that generates almost photo-realistc people who seem to have come straight out of your favourite Disney/Pixar/Dreamworks animated movie.

https://www.twitter.com/Norod78/status/1297513475258953728

How to make a photo-realistic cartoon model #

So what's going on here? This blended network is actually the product of a rather convoluted process which I'll try and briefly explain.

Transfer learning #

We start with the classic StyleGAN model which is trained on photos of people's faces. This was released with the StyleGAN2 code and paper and produces pretty fantastically high quality results.

Not everyone has multiple-GPUs and weeks of time to train a model so a shortcut lots of people is called transfer learning, where they take this "pre-trained model" and then train it on some new data. This gives good results really quickly, and even more so if the new dataset is some also of faces.

Doron fine-tuned the faces model on a dataset of various characters from animated films. It's only around 300 images but enough for the model to start learning what features these characters typically have.

Once the model is trained just a little bit it gives outputs that look like below

The output is OK for such a small amount of training on a small dataset, it's clearly got the big eyes thing down pretty well. The problem is that the style of images in the dataset is a bit of a mish-mash, some are CG some are hand drawn, and lots are quite low resolution. The model tries to replicate all these things and comes off worse for it.

Blend the models #

But as this model was fine tuned from the original faces model, we can perform a trick where we directly swap parts of the models around. This gets interesting because, due to the structure of StyleGAN, different layers in the model affect the appearance in different ways. So low resolution layers affect the pose of the head and shape of the face, while high resolution layers control things like lighting and texture. When Doron used my layer swapping script (Colab here) to take the high resolution layers from the original model, and the low resolution from his fine-tuned cartoon model. You end up with a hybrid which has the structure of a cartoon face, but photo realistic rendering!

If you generate images using the original faces model and the blended cartoon model, you can see how there is a clear relationship between the two, the identities appear to be the same, but the features have been shifted to give them a cartoon look.

Now toonify yourself #

These StyleGAN face models can produce a huge diversity of faces and it's actually possible to find basically any face inside the model. It's actually a straight forward process to search for any image of a face in the model. So given an example image you want to find you can find a "code" (aka latent vector) which, when given as an input to the model, will produce an output which looks almost exactly like the face you're looking for. Below the original is on the left, and the generated image on the right (it's hard to tell right!?)

Now you have a code that represents a face, you can give this as input to the blended model and given that they are closely related, you will get the same face, but modified to look like a "toon" version!

If you want to try this process yourself you can use this colab notebook.

BTW all credit to Doron for making the model and showing this self toonification works wonderfully!

Next steps #

I followed up this stuff by making a website where anyone could Toonify themselves in a few seconds, it got pretty popular (too popular!). See the details of how it was made.

More examples #

StyleGAN network blending

2020-08-25T00:00:00Z

Making Ukiyo-e portraits real #

In my previous post about attempting to create an ukiyo-e portrait generator I introduced a concept I called "layer swapping" in order to mix two StyleGAN models[^version]. The aim was to blend a base model and another created from that using transfer learning, the fine-tuned model. The method was different to simply interpolating the weights of the two models[^interpolation] as it allows you to control independently which model you got low and high resolution features from; in my example I wanted to get the pose from normal photographs, and the texture/style from ukiyo-e prints[^style-transfer].

The above example worked ok, but after the a recent Twitter thread popped up again on model interpolation, I realised that I had missed a really obvious variation on my earlier experiments. Rather than taking the low resolution layers (pose) from normal photos and high res layers (texture) from ukiyo-e I figured it would surely be interesting to try the other way round[^texture].

https://vimeo.com/451284388

It was indeed interesting and deeply weird too! Playing around with the different levels at which the swap occurs gives some control over how realistic the images are. If you've saved a bunch of network snapshots during transfer learning the degree to which the networks have diverged also give some interesting effects.

You also see some wonderfully weird effects because of the fact that ukiyo-e artists almost never drew faces straight on. As the original faces model is mostly straight-on the model has a somewhat tough time adapting to this change.

Blending other StyleGAN models #

You can also swap models which are trained on very different domains (but one still has to be fine-tuned from the other). For example the Frea Buckler model trained by Derrick Schultz. Swapping out the original FFHQ trained model into this one is in a sense replacing the rendering to be of the faces model, but the structure to be from the new one.

https://vimeo.com/451291240

As well as being pretty mesmerising it seems to give some hints as to how the model transferred domains. It looks like it's mostly adapted features corresponding to the background of the original images to serve as the structural elements of the new model. Many of the new images look like an "over the shoulder" view point, and the original faces have been pushed out of frame (although as noticed they are still lurking there deep in the model). Although you could probably understand these details of the mechansim of domain transfer using some statistical analysis of the network weights and internal activations, this is quite a simple and pretty way of getting an intuition.

Get the code and blend your own networks #

I've shared an initial version of some code to blend two networks in this layer swapping manner (with some interpolation thrown into the mix) in my StyleGAN2 fork (see the blend_models.py file). There's also an example colab notebook to show how to blend some StyleGAN models[^models], in the example I use a small faces model and one I trained on satellite images of the earth above.

I plan to write a more detailed article on some of the effects of different blending strategies and models but for now the rest of this is documenting some of the amazing things others have done with the approach.

Over to you StyleGAN Twitter! #

Disneyfication #

Shortly after sharing my code and approach, some of the wonderful StyleGAN community on Twitter started trying things out. The first really amazing network blend was by Doron Adler, he mixed a model fine-tuned on just a few images of Disney/Dreamworks/Pixar characters to give these uncannily cartoonish characters.

https://www.twitter.com/Norod78/status/1297513475258953728

He also used a StyleGAN encoder to find the latent representation of a real face in the "real face" model then generate an image from the representation using the "blended" model with amazing semi-real cartoonification results:

https://www.twitter.com/Norod78/status/1297849293299212288

I think this approach would make a great way of generating a paired image dataset for training a pix2pixHD model, i.e. the StyleGAN Distillation approach.

Resolution dependent interpolation: foxes, ponies, people, and furries #

It was originally Arfa who asked me to share some of the layer swapping code I had been working on. He followed up by combining both the weight interpolation and layer swapping ideas, combining a bunch of different models (with some neat visualisations):

https://www.twitter.com/arfafax/status/1297694374470402055

The results are pretty amazing, this sort of "resolution dependent model interpolation" is the logical generalisation of both the interpolation and swapping ideas. It looks like it gives a completely new axis of control over a generative model (assuming you have some fine-tuned models which can be combined). Take these example frames from one of the above videos:

On the left is the output of the anime model, on the right the my little pony model, and in the middle the mid-resolution layers have been transplanted from my little pony into anime. This essentially introduces middle resolution features such as the eyes and nose from my little pony into anime characters!

Parameter tuning #

Nathan Shipley made some beautiful experiments trying to get the Toonification effect just right by adjusting two of the key parameters: the amount of transfer learning to apply (measured in thousands of iterations) and the resolution layer from which to swap. By tuning these two you can pick out just the degree of Toonificaiton to apply, see this lovely figure made by Nathan:

Then he applied First Order Motion model and you get some pretty amazing results:

https://www.twitter.com/CitizenPlain/status/1308824021803372549

Going further #

I think there's lots of potential to look at these blending strategies further, in particular not only interpolating between models dependent on the resolution, but also differently for different channels. If you can identify the subset of neurons which correspond (for example) to the my little pony eyes you could swap those specifically into the anime model, and be able to modify the eyes without affecting other features, such as the nose. Simple clustering of the internal activations has already been shown to be an effective way of identifying neurons which correspond to attributes in the image in the Editing in Style paper so this seems pretty straightforward to try!

[^version]: I did all this work using StyleGAN2, but have generally taken to referring to both versions 1 and 2 as StyleGAN, StyleGAN 1 is just config-a in the StyleGAN 2 code.

[^interpolation]: Interpolating two StyleGAN models has been used quite a bit by many on Twitter to mix models for interesting results. And as far as I'm aware the idea first cropped up in generative models in the ESRGAN paper.

[^style-transfer]: If this sounds a little bit like style-transfer then you're not far off there are some similarities. In face StyleGAN's architecture was inspired by networks designed for style transfer.

[^texture]: It seems to be a general rule that neural networks are better at adding texture than removing it.

[^models]: If you're in search of some models to blend then see collection of pretrained stylegan models (I intend to add a field as to which ones have been fine-tuned in the near future.)

From feature space to input space

2020-08-05T00:00:00Z

Input space and Feature space are different sides of the looking glass #

Neural networks (for image tasks) take in images and put them through a series of, typically convolutional, transformations. The world of those input images consisting of red, green, and blue pixel values we can call input space and the intermediate values that get computed in the network, the transformed versions of the images, we might call feature space.

There are many different levels of feature space, one for each layer in the network, and there is much work to try and understand what the values and dimensions in this space represent[^feature-vis]. But fundamentally these internal representations computed by the network just look like images, ones with (generally) many more channels that the three colours we are familiar with.

TODO find nice diagram of input space and feature space in the network

How is Feature space just like Input space? #

As pointed out in the original batchnorm paper we can consider any portion of a network as a "sub-network" which takes in some activations from the previous layers and applies transforms to this. In this sense the previous layer's feature space is just the input space for the later layers.

Given that there is such a strong connection between input and feature space, it makes sense that anything we can do to improve the training process of the network in one space might be useful to also do in the other.

Batch normalisation is just input normalisation in feature space #

It has been long known that the network training converges faster if its inputs are whitened – i.e., linearly transformed to have zero means and unit variances, and decorrelated. As each layer observes the inputs produced by the layers below, it would be advantageous to achieve the same whitening of the inputs of each layer.[^ioffe2015batch]

It is standard practice to normalise the inputs of a machine learning model, at least ensuring the inputs have mean zero and a standard deviation of 1. The logic which inspired batch normalisation is that if such an operation is beneficial in input space, it is logical to expect it to also be of benefit in feature space.

Convergence is usually faster if the average of each input variable over the training set is close to zero. [^lecun-backprop]

The invention of batch normalisation was extremely important important in deep learning, it allowed efficient training of increasingly large networks at higher learn rates and has become standard practice in many neural network architectures. Since the original paper's discussion of how batchnorm helps by reducing "internal covariate shift" there has been some confusion as to how exactly it works, but for the best discussion I've seen, look at this Twitter thread by David Page:

https://twitter.com/dcpage3/status/1171867587417952260

Clearly, adapting the concept of input normalisation to feature space is not trivial, and batch normalisation requires various new mechanisms and non-obvious assumptions to work (e.g. that batch statistics are sufficient, learned scale and offset parameters, and changes in behaviour between train and test time). In fact many of these unusual features of mean that Batch Normalisation turns out to often be a problematic layer.

Dropout works in feature space #

This pattern of taking concepts from one space and translating them to the other has also been applied to the regularisation technique of dropout. In dropout we randomly set the activations of some neurons to zero during training. It was popularised by use in AlexNet and for a while was extremely common practice in deep learning. The fact that this rather extreme procedure of setting (often significant) amounts of the feature space pixels to zero should actually work, let alone be beneficial, is not at first glance obvious (at least not to me), but there are various intuitive explanations for why this should be.[^thinking-about-dropout]

Nowadays dropout is less prevalent than it used to be. This would seem to be in large part due to the fact that it was predominantly used in the final fully connected layers of networks which are no longer common. It doesn't work so well for convolutional layers, but this is something we'll come back to later.

Cutout is dropout in Input space #

So if dropout was a successful approach for regularisation which works by zeroing pixels in feature space, can this be translated to input space? One problem is that dropout only seems to work well for fully connected layers and not so much for convolutional ones.

While dropout was found to be very effective at regular-izing fully-connected layers, it appears to be less powerfulwhen used with convolutional layers ...

... neighbouring pixels in images share much of the same information. If any of them are dropped out then the information they contain will likely still be passed on from the neighbouring pixels that are still active. [^cutout]

That same reasoning is also going to apply to images, but like we saw before porting these ideas from one space to another is never quite straightforward (if it was, it would have been done already).

DeVries and Taylor introduced a data augmentation method called Cutout in the paper "Improved Regularization of Convolutional Neural Networks with Cutout" where they set random rectangles of the input images to be zero, and as they point out:

Cutout can be interpreted as applying a spatial prior to dropout in input space [^cutout]

The spatial prior they mention above is the fact that they remove a contiguous patch of the input images (they use a rectangle but state that the shape is not important) rather than zeroing random pixels as in dropout.

One other difference here is the difference in training and test time weight scaling that is used in dropout is no longer required. For the input space we centre our data around zero (which is not true in feature space), so as long as we fill our patches zero won't affect the expected bias.

Feature space to input space and back again... #

In fact those same modifications to dropout that made it work for input space can be ported back to feature space to solve the problem we pointed out earlier: dropout doesn't work well for convolution layers.

The Dropblock paper shows how the idea of zeroing out a contiguous region allows us to use dropout it feature space effectively. As the authors state "DropBlock generalizes Cutout by applying Cutout at every feature map in a convolutional networks."[^dropblock]

Unfortunately getting dropblock to work effectively seems to require hyper-parameter tuning, in terms of block size and where it is applied, as well as tweaking training by gradually increasing the amount of drop. The paper also states that "Although Cutout improves accuracy on the CIFAR-10 dataset ... it does not improve the accuracy on the ImageNet dataset in our experiments".

What other innovations can be made by generalising ideas between spaces? #

So that's a couple of examples of how generalising ideas which were successful in one space to the other space can lead to useful innovations in deep learning, and there are more like MixUp and Manifold MixUp.

What others concepts could be transferred from one space to the other? One possibility is the use of image transforms for data augmentation. Random horizontal flips, or small amount of resizing are commonly applied to input images. Could the same be applied to feature space? One complication, particularly for horizontal reflection, is that pixels in feature space are likely to have some directionality that pixels in image space do not. So maybe random rescaling is a better place to start? But as we've seen above, methods often need some modification to be relevant in the other space.

In fact we can also look for other "spaces" to apply our ideas. For example the parameters of the network itself can be considered to be a high-dimensional weight space. And although the properties of this are likely to be somewhat different to feature or input space, perhaps some of the ideas can also transition.[^weight-standardisation]

Image credits #

[^feature-vis]: For beautiful visuals and great insights into what this feature space represents see the fantastic work on feature visualisation and the follow ups by Chris Olah.

Olah, Chris, Alexander Mordvintsev, and Ludwig Schubert. ‘Feature Visualization’. Distill 2, no. 11 (7 November 2017): e7. https://doi.org/10.23915/distill.00007.

[^ioffe2015batch]: Ioffe, Sergey, and Christian Szegedy. ‘Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift’. ArXiv:1502.03167 [Cs], 2 March 2015. http://arxiv.org/abs/1502.03167.

[^cutout]: DeVries, Terrance, and Graham W. Taylor. ‘Improved Regularization of Convolutional Neural Networks with Cutout’. ArXiv:1708.04552 [Cs], 29 November 2017. http://arxiv.org/abs/1708.04552.

[^thinking-about-dropout]: There are various different ways to interpret how dropout is working: simply adding noise, approximating averaging an ensemble, or prevent co-adaptation. The original paper outlines these and it would be interesting to revist what evidence there is for what intuition is correct.

[^lecun-backprop]: LeCun, Yann A., Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. ‘Efficient BackProp’. In Neural Networks: Tricks of the Trade: Second Edition, edited by Grégoire Montavon, Geneviève B. Orr, and Klaus-Robert Müller, 9–48. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, 2012. https://doi.org/10.1007/978-3-642-35289-8_3.

[^dropblock]: Ghiasi, Golnaz, Tsung-Yi Lin, and Quoc V. Le. ‘DropBlock: A Regularization Method for Convolutional Networks’. ArXiv:1810.12890 [Cs], 30 October 2018. http://arxiv.org/abs/1810.12890.

[^weight-standardisation]: Qiao, Siyuan, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. ‘Weight Standardization’. ArXiv:1903.10520 [Cs], 25 March 2019. http://arxiv.org/abs/1903.10520.

About this site 🏡

2020-07-19T00:00:00Z

This is old, since then I gave up the pain of Gatsby and updated the site to run on 11ty

I've had many different versions of a home page over the years. I've left traces on lots of different social media sites (but only really use Twitter now), kept a food blog[^recipes], posted about generative art on a wordpress blog, even written my own simple static site generator (for no apparent reason).

This is my latest attempt (and I'm sure it'll stick this time, honest...). Hopefully this will be a place for me to share some of the more interesting things I make and do and even if it's not much interest to anyone else, hopefully I'll like it. Keep up to date with the RSS feed.

The site is built with Gatsby and will give me somewhere to play with web technologies and learn a little more JavaScript and React. Under the hood I'm writing posts in MDX so I can include slightly more interactive content that in plain old markdown blogs. I've enjoyed seeing some of the recent activity around digital gardens and the indie web and hope to incorporate some of those ideas here too.

Gatsby has impressed me with its ease of use and adaptability, it's made making websites fun again.

Making use of these great things #

Gatsby
MDX - for writing posts
MDX embed - for embedding tweets and other things
Leaflet - for displaying big images
gatsby-plugin-react-leaflet - for working with leaflet
Littlefoot - for foot note pop overs[^littlefoot]
TinyDB, pysftp, imageio, and Pillow for managing my stream (now dead)
Leaflet EdgeBuffer

[^littlefoot]: Getting Littlefoot to work correctly on the Gatsby production build actually requires something a little different to that described in the readme, see this issue.

[^recipes]: I still keep some recipes at https://recipes.justinpinkney.com but these are just for me and that food blog is long dead.

CakeGAN 🍰

2020-07-14T00:00:00Z

These cakes are a lie, they're all generated by a neural network.

Random fake cakes are nice, but eventually my grand idea is to make a bespoke cake designer (I'm in this for the latent space you know!). Before I can do that I need to train a model, let's call it CakeGAN. So here is a delightful recipe for how to bake your very own non-existant cakes.

How to bake a CakeGAN #

Ingredients #

Data - 40,000+ images of cakes of all varieties cut into equal bite-sized pieces
Architecture - StyleGAN 2
Code - My StyleGAN2 fork
Compute - A powerful GPU (Google Colab will do just fine)

Method #

Collect your data #

Scraping an image search engine is a pretty easy way of assembling a pretty huge, but low quality dataset. I used this Bing image search scraper[^1] to grab a bunch of images for every search term I could think of relating to cakes. After a little time I had many GBs of (mostly) cake images on my hard drive.

That's the raw material assembled, but it's going to need some refinement first.

Prepare your data #

The images you get back will probably be a pretty mixed bag

In their raw form they're not really suitable for training StyleGAN, in particular the images need to be a fixed aspect ratio (I'm going to go for square). It's a generally accepted piece of Deep Learning Folklore that StyleGAN gives better quality results when trying to generate images which have centrally placed objects[^2].

Conveniently there are a wealth of pre-trained cake detectors available! They aren't usually called that, but it happens that "cake" is one of the categories in the COCO object detection challenge, so any COCO detector can find cakes. Using the best model from Detectron2 makes this a pretty easy task.

Then sprinkle on a bit of padding, cropping, and resizing as required and that's the main ingredient ready to go, here's what it should look like:

Train your model #

Next it's time to start baking your model in a warm GPU heated oven. The model I'm training is 256x256 (config-e) which is a bit lighter weight than a the full fat 1024x1024 (config-f) model primarily used in the paper.

I use Google Colab for training, and use a fork[^3] of Skyflynil's StyleGAN2 repo which allows you to keep the training data as jpegs (stuffed inside tfrecords), this keeps that dataset size now to a reasonable number of GBs.

If you get a P100 via Colab the training process takes about 9 minutes per tick[^4] (make sure you optimise your GPU memory usage for the smaller model by setting the minibatch_gpu_base to 16). Calculating the FID takes around 10 minutes and I think is worth it to keep track of progress.

Overall I trained for around 5 million images (which works out around 6 days of training) achieving an FID of 13.6. The loss and FID curves are shown below, and you can see that I could have probably kept on training (the FID is still going down and there doesn't seem to be much sign of the discriminator over fitting[^5]) but the gains seemed pretty marginal at that point.

You'll know when your CakeGAN is fully baked when the FID no longer improves, or a skewer inserted comes out clean or it springs back lightly when poked. Here's a small selection of generated cakes.

And below is a zoomable grid arranged using feature extracted using an imagenet pretrained CNN projected into 2D using UMAP.

You can download the final trained model from my Awesome Pretrained StyleGAN 2 repo.

Serving #

Once we've the model has cooled it should be serving up some delicious interpolation videos. But the best part of a well baked GAN is the smooth and creamy latent space. For a start StyleGAN is famous (in part) for its style mixing capabilities, so let's mix some cakes.

If I want to make a cake editor I need to try and find meaningful directions in the latent space. I've got a dataset with some noisy labels I could try and use, but for a first pass using GANSpace is a quick and easy way of trying to automatically find meaningful directions[^ganspace-notebook].

Some of the directions found by GANSpace seem promising, there are some hints of cake to slice, or more chocolate or fruit topping vectors. But the results below are very cherry picked and these don't seem to generalise terribly well.

[^1]: This scraper is based on a Google image scraper which now seems to have been broken by changes in the Google images search page.

[^2]: This may be due to the fact that StyleGAN uses a learned constant as the input to the convolutions, but I'm not aware of any actual evidence for this, (it's more deep learning hearsay than folklore).

[^3]: There are so many different forks of the original StyleGAN2 repo that it is a bit of a nightmare. If someone was feeling adventurous they could try and collect all the features into one. The closest I know for this is Peter Baylies fork, but last time I tried to use it, it was broken 🤷.

[^4]: 1 tick corresponds to 6000 images

[^5]: To check this I'd like to use the heuristic presented in "Training Generative Adversarial Networks with Limited Data" which is the average value of the sign of D(real), but I don't have access to this data.

[^ganspace-notebook]: I've been using this handy Colab notebook from @realmeatyhuman for running GANSpace

Big images with Leaflet 🗺️

2020-07-13T00:00:00Z

This is all old. Since then I've updated my blog to run on 11ty and life is much simpler.

Below is a big image produced by my neural network feature visalisation library Sumie please zoom in.

This page is mostly just a test of the integration of Leaflet for displaying big images on my Gatsby powered homepage. Below are some brief scratchings on how this works so I don't forget.

What is MDX actually good for? #

I recently moved my website over to MDX as I was tempted by the prospect of adding more interesting and interactive content that is normally in plain old markdown blogs. MDX lets you include react components directly into your markdown syntax documents and render them to a static page with Gatsby. I've wanted to put large zoomable images into some of my posts and this seems like a straightforward way.

Make tiles from a big image #

Some of my generative art or visualisation algorithms make images in the range of a few hundred megapixels, displaying these over the web takes a little care. The first step in translating a single giant image into a interactive zoomable is to slice it up into a bunch of tiles at different resolutions. This Python repo provides all you need for this step.

import deepzoom

creator = deepzoom.ImageCreator(
    tile_size=256,
    tile_overlap=0,
    tile_format="jpg",
    image_quality=0.8,
    resize_filter="bicubic",
)

creator.create("input.tiff", "output/zoomable.dzi")

Some code like the above will convert a large tiff file into a folder containing subdirectories with many tiles at different resolutions, which I can then upload to my web host.

Display with leaflet #

Leaflet then provides a pan and zoomable display of the image. Leaflet is normally used for displaying maps, but is simple to adapt it to using the tiles generated above by specifying the correctly formatted url, and setting the co-ordinate reference system to "simple", i.e. a plain grid.

One small issue with the above is that Leaflet doesn't like fractional tiles so there are currently some weird edge effects which I could solve by making sure I pad all the tiles to the full dimensions.

Incorporating leaflet into a Gatsby site is happily very simple[^1] thanks to the Gatsby React-Leaflet plugin which takes care of properly wrapping up the existing React-leaflet library (which itself makes Leaflet accessible as React components). Writing the react component required to display the image is very straightforward and then I can directly write the following in my markdown file to give the zoomable image at the top of the page.

<BigImage
    options={ {center:[-0.25, 0.35], zoom:10, minZoom:9, maxZoom:14 } }
    tile_url="http://assets.justinpinkney.com/sandbox/montage/montage_files/{z}/{x}_{y}.jpg" />

One simple gotcha is highlighted on gatsby-plugin-react-leaflet's readme. Make sure any usage of leaflet or react-leaflet are wrapped in a check that the window is defined. During build time the plugin will stub out the leaflet loader so any imports will return an undefined, and if you try and use these during build time (rather than run time, which is what the window check above ensures) you are likely to get x is undefined type errors.

[^1]: I actually started off trying to use OpenSeadragon which has a much smoother pan and zoom action, but with no existing Gatsby integration it seemed like the harder option to start with.

Awesome Pretrained StyleGAN

2020-07-03T00:00:00Z

I maintain two collections of links to StyleGAN models pre-trained on a variety of datasets:

Most of these have been shared via the very active StyleGAN creative community on twitter, and if you're aware of any others then please send them my way. Either create an issue or fill out one of the following forms:

Pretrained models are useful for lots of things #

Speed up model training with transfer learning #

Apart from just generating some example image of whatever the model was trained on, pre-trained models are a super useful for training your own model using transfer learning.

If you want to train a model which is similar to an existing one, for example Ukiyo-e faces then taking a pre-trained model as your starting point to get you some decent results within just a few hours rather than days of training time.

Get weird results with intermediate models and mixing #

Transfer learning (aka fine-tuning) also sometimes gives you some pretty interesting results just as the model starts to transform the generated images from the original objects to the new thing you're training for. See a couple of nice examples below.

https://www.twitter.com/Norod78/status/1255200236181630979 https://www.twitter.com/mmariansky/status/1226756838613491713

In fact you can go further and combine two models, one which has been fine-tuned from another and either do weight averaging or layer swapping to effectively mix the outputs.

Do experiments and investigate the properties of a latent space #

Pre-trained models can also be useful if you want to investigate the properties of specific modifications and manipulations of a trained GAN. In fact my Awesome StyleGAN made and appearance in the excellent GANSpace paper.

A well trained model is also useful if you just want to mess around with the internals of StyleGAN,

Maybe transfer learning will become the standard way to train a GAN #

Transfer learning is currently a very active area of research in the world of GANs, in particular there have been a bunch of publications recently looking at methods for preventing the discriminator from over-fitting when you only have a small dataset. These would seem to make it possible to get very good results using a pre-trained model and only a small amount of data.

Contribute! #

If you have a StyleGAN model you'd like to share I'd love it if you contribute to the appropriate repository. In particular I see lots of users of RunwayML sharing links to their models on Runway, but not sharing the .pkl files, please set those models free!

MATLAB StyleGAN Playground 🙃

2020-06-18T00:00:00Z

Everyone who's ever seen output from GANs has probably seen faces generated by StyleGAN. Now you can do the same in MATLAB!

StyleGAN (and it's successor) have had a big impact on the use and application of generative models, particularly among artists. Much of this has been a combination of accessible and (fairly) straightforward to run code, great stability in training, a particularly well formed and editable latent space representations, and ease of transfer learning. It's been taken up very readily by people on twitter and tools like runwayml make it very accessible for non-programmers too. There's also a wealth of pre-trained models available, and I maintain my own collection of them.

One thing that makes StyleGAN less appealing is that they're written in TensorFlow. And although they're very performant it makes them much more less accessible to modify and adapt. In fact it's a telling reflection on the differences between deep learning frameworks that many subsequent papers proposing modifications and extensions to StyleGAN have actually chosen to use a PyTorch port rather than the TensorFlow original.

StyleGAN in MATLAB (yes really) #

Since R2019b MATLAB has had support for a low-level interface to deep learning. Rather than dealing with layers, you can write networks as normal functions. This makes it a good deal easier to port things from other frameworks (in particular PyTorch).

So as an exercise in testing out the new features and really getting to grips with the details of StyleGAN, I ported the original to MATLAB. Below is a version of the classic style mixing figure from the original paper (using the original FFHQ model)

To use a pre-trained generator I have a small bit of Python code (which you can run freely in Colab without needing any Python setup) which allows you to take a model trained in the original implementation and convert the weights to a .mat file for use in MATLAB. If you're looking for some pre-trained models a good place is my pretrained StyleGAN model collection.

As well as simply creating faces using the generator it's now very simple to use the rest of the deep learning features in MATLAB. For example you can use one of the pre-trained models to compute a perceptual path length to see the differences in interpolation methods. (Note the original uses a different perceptual loss so the numbers aren't comparable to those in the paper) Interpolation in z space makes some random glasses appear, whereas using w space gives a much smoother transition.

Hacking the Model #

The big advantage (at least for me) of having an implementation in MATLAB, rather than TensorFlow, is that it becomes a lot easier to monkey around with the internals of the model.

Currently there is a system of callbacks so that you can reach into the model and mess around with things like the internal activations. Here's an example where you can modify the learned constant that is at the begning of the model to change the content of the generated image. If you extend it you can create more background (or endless hair), if you set it to some random numbers you get a mish mash of facial features.

And here's another where I'm using just rotating the activations at a particular layer. If you're interested in what can be done with this sort of approach, see the work by Terence Broad which takes the idea a lot further.

No Training #

Unfortunately MATLAB doesn't yet support double differentiation so it's not possible to train a StyleGAN model in MATLAB.

The Code #

The StyleGAN implementation is released as an open-source project on GitHub.

The code is currently in a state that it's probably not super comprehensible to someone else (there are a few example live scripts at least). If anyone is actually interested in using this, create an issue or reach out to me on Twitter and I'll endeavour to get round to making it a bit more useable.

Of course shortly after I'd done all of this Nvidia went ahead and released StyleGAN2 which redefined the state of the art again!

And finally #

If you want a peek at what things I've used this for I've collected some of my experiments with generated faces is this Twitter thread:

https://www.twitter.com/Buntworthy/status/1275175544087367682

Ukiyo-e Yourself with StyleGAN 2

2020-06-10T00:00:00Z

I've spent some time training a StyleGAN2 model on ukiyo-e faces. Here are some results from training and some experimentation with model interpolation.

Dataset #

I've spent some time collecting face images from museum images, suprisingly AWS Rekognition does a reasonable job of detecting faces and landmarks. As most of the images provided by museums are not very high resolution I've also used ESRGAN to upscale to a final 1024x1204 resolution.

Now I've got a few thousand high-resolution and algined ukiyo-e faces ready for StyleGAN training

Training results #

I'm fine tuning from the original FFHQ model (config-e) which has lots of frontal portraits, but these were very uncommon in uikyo-e pictures. The model quickly learns the basic style; then spends a while with a dilemma of how best to turn people's heads; then settles on one side.

Eventually the model basically learns not to have frontal faces and interpolations generally give sharp transitions from left to right looking faces

Ukiyo-e yourself #

The early stages of training are interesting in that you often see "ukiyo-e ish" looking faces but with traits of the original. Take a look at this example of the same seed for different iterations in training, in the 2nd panel he's clearly still wearing a suit!

This early stage of training seems like it might be a promising way of creating a "real" to "ukiyo-e" face model, but at this point the model is already starting to twist the pose of the faces.

Model interpolation #

Playing with well known technique of averaging the weights of the base and the transferred model helps get a bit closer to the original, but the pose is still way off. Here's a few frames of interpolation from the weights of one model to another

Layer Swapping #

So I've also been playing with something you could call "layer swapping". Taking different resolution layers from the different models and combining them. This helps to avoid changing the pose (which is controlled by early, low resolution layers). Here's an animation as I progressively swap in more layers from the original FFHQ model into the ukiyo-e model.

Swapping in only the lower resolution layers from FFHQ into Ukiyo-e serves to preserve the pose of the generated face, but still transfer the features and style of a typical ukiyo-e portrait. Here's a detail of the point at which I think it looks best.

There are still some artifacts but I think it's a fun technique and sometimes gives some nice results. Here are a bunch more cherry picked examples:

Since this original example I have put a bit more work into testing out these layer swapping ideas. You can get code to do it yourself, and see what other weird and wonderful things myself and others have come up with in this post.

Other thoughts #

I tried using CycleGAN to iron out the defects in the outputs of the combined model. Training it to convert from the modified model to the original ukiyo-e faces, but the brief attempts I've given haven't got any good results:

While I'm at it, a CycleGAN trained to turn Ukiyo-e faces into real ones does give some pretty amusing results:

Currently my "layer swapping" is just a straight copy of the weights of certain layers of one model to the other. But this leads to a harsh transition in the model. Maybe a smoother transition would give nicer results, for example gradual interpolating from one model to the other as you move through the layers.

Another completely different idea would be to simply not train the lower resolution layers to begin. This might prevent the model changing the low resolution appearance (like pose) while getting the appearance of ukiyo-e right

pix2pixHD model distillation #

So one thing to remember is that the people above are not real people, they're generated by the FFHQ trained model. To apply this to a real person's face we'd first have to find their latent representation using an optimisation process (like the projector included in StyleGAN2). To skip out on this laborious step I tried to train a pix2pixHD model (as in the paper StyleGAN2 Distillation for Feed-forward Image Manipulation) to "distill" the transformation. This way I can give it an arbitray image of a person and get the ukiyo-e version. The end results aren't too bad

But there are still some interesting failure cases when the input image isn't aligned in the way the training images are.

MATLAB pix2pix 🏢

2020-06-09T00:00:00Z

Of all the GAN architectures pix2pix is a personal favourite. It popularised the use of GANs for image to image translation, it's nice and simple, trains relatively quickly, and invariably produces some surprisingly pleasant results. If you don't know what pix2pix is, see the original project page which has some nice examples and a demo. Or just search Twitter for #pix2pix for some fun examples.

Since R2019b MATLAB has had support for automatic differentiation and it's been possible to train GANs. So one of the first things I tried out was writing a MATLAB version of pix2pix (unlike Python it's easy to be the first to write and implementation of a deep learning paper in MATLAB).

The classic example for pix2pix is the facades dataset and running this is gives similar results to the original paper (although training time is quite a lot slower).

I've released this as an open-source project on GitHub if you run into any problem please file an issue.

Colour Sorter 🌈

2020-06-08T00:00:00Z

Takes a picture and rearranges the pixels to make another.

A little generative art Java application I made a long time ago which implements an algorithm to take all the pixels from an image and use them to make a new image, based on a (sort of) sorting algorithm.

https://youtu.be/FCp3UCBGCmc

This was all inspired by József Fejes's all RGB sorting work. I implemented his basic concept for pixel colour sorting, but using an image as the source of pixels and added lots of options for sort order, distance metrics, and starting patterns. The combination of options as well as the sheer variety of possible starting material (i.e. any image or photograph) produces a huge variety of results which is very fun to curate.

You can find the source code on GitHub.

Examples #

My Day Job 🧑‍💻

2020-06-07T00:00:00Z

Midjourney #

I work as a machine learning researcher at Midjourney, working on state of the art text-to-image models.

Before #

Lambda Labs - I worked as a Senior Machine Learning Researcher at Lambda Labs mostly with a focus on computer vision and generative models.

MathWorks - I used to work as a senior consultant at MathWorks helping people to: develop computer vision and deep learning algorithms, write better MATLAB code and embrace good software development practices, develop and deploy more complex application than your average MATLAB user usually thinks about.

As part of that job I was able to release some open source projects:

Sagentia - I used to work at a product development consultancy called Sagentia.

PhD - Before I had a real job I did a PhD in Biophysics at the University of Oxford.

MATLAB Face Detection with MTCNN 🔎😄

2020-06-06T00:00:00Z

Get a fast and accurate face and facial feature detector for MATLAB here

Intro #

Everyone pretty much takes good quality face detection for granted these days, and it's essentially a solved problem in computer vision. Everyone's photo app can detect all the face you care about and computer vision competition results are increasingly focussing on performance for difficult, small, or occluded faces.

Although MATLAB has a face detector built into the Computer Vision Toolbox I'm sure most who have had some reason to use it have come away disappointed. Honestly, it's pretty outdated as deep learning has totally revolutionised computer vision since that face detector was released.

Enter MTCNN #

There are now tons of deep learning based face detectors, each of them eeking out more and more performance on standard benchmarks. But sometimes you just want something that is pretty good and well tested in real life.

Multi-task Cascaded Convolutional Neural Network (MTCNN) is a little old but has a fairly simple architecture, is small and fast, and performs well. It also has the additional advantage of outputting the locations of facial features (eyes nose and mouth). It's been widely used for a pre-processing step in lots of other applications and it works well and reliably.

MTCNN in MATLAB #

I've ported the original MTCNN pre-trained weights into MATLAB, using some of the deep learning features introduced in R2019b. (I've also done some work to make sure that it still runs in R2019a, although it's a little slower.)

I've released this as an open source project, the code and toolbox (for simple install) is available on GitHub

Training? #

One thing to note is that the current code doesn't support training a new model, although this would be perfectly possible to do. People don't seem particularly interested in training a new model, just using the standard pre-trained weights. If I'm wrong about this and you're desperate to train you own models please comment on this issue to let me know.

Biophysics 🔬🧬

2020-06-04T00:00:00Z

My biophysics phd was focused on single molecule fluorescence methods for studying DNA-protein interactions.

A bit more specifically I used single molecule methods such as FRET (Forster resonance energy transfer), PIFE (protein induced fluorescence enhancement), and a method I developed, TFM (tethered fluorophore motion), to study a range of DNA processing proteins (DNA polymerase, recombinases, and motor proteins).

This was where I first started to write code in earnest, learnt the appeal of building tools and methods, started processing images, and got to play with lasers.

Here are the papers I published back then:

[Capturing reaction paths and intermediates in Cre-loxP recombination using single-molecule fluorescence](Capturing reaction paths and intermediates in Cre-loxP recombination using single-molecule fluorescence.pdf)
[Conformational transitions during FtsK translocase activation of individual XerCD–dif recombination complexes](Conformational transitions during FtsK translocase activation of individual XerCD–dif recombination complexes.pdf)
[Extending and combining single molecule methods to study site specific recombination](Extending and combining single molecule methods to study site specific recombination.pdf)
[Rotavirus mRNAS are released by transcript-specific channels in the double-layered viral capsid](Rotavirus mRNAS are released by transcript-specific channels in the double-layered viral capsid.pdf)
[Tethered Fluorophore Motion Studying Large DNA Conformational Changes by Single-fluorophore Imaging](Tethered Fluorophore Motion Studying Large DNA Conformational Changes by Single-fluorophore Imaging.pdf)

Better OBJ model loading in Processing

2015-04-12T00:00:00Z

Processing is a great at many things, it's so easy to do so much, including displaying things in 3D. Still it's easy to make things grind to a halt when trying to do to much. But there are a lot of things you can do to speed things up like PShape recording, using Vertex Arrays to display lots of points, or writing GLSL Shaders.

Loading a .obj model in Processing is as easy as using loadShape(), but loading a reasonably complicated textured obj models things slow down dramatically. Things seem to run happily when there are no textures involved. I think it's something to do with the fact that each face of the model is loaded as an individual PShape, that are all lumped together (but it's for a good reason).

You can pretty easily speed things up by taking the PShape made by loadShape and grabbing all the child shapes and putting them into a single PShape. You can't mix quads and triangles in a single PShape, and different textures and material properties would need to be split up too. Still from a crawling 4 fps, we easily get back up to 60 fps. Which is cool, hopefully that's helpful to someone, as it took me ages to figure out why things were going so slowly. And at some point I might have a crack at writing my own version of the OBJ loader.

PShape objShape, triShape, quadShape;

void setup() {
  size(720, 360, P3D);
  // Load the obj model normally
  objShape = loadShape("heartDecim.obj");

  // Make a PShape with the all the faces with three vertices
  triShape = createShapeTri(objShape);
  // Make a PShape with the all the faces with four vertices
  quadShape = createShapeQuad(objShape);
}

public void draw() {
  background(255);

  translate(width/2, height/2, 0);
  scale(20);
  rotateX(PI);
  rotateY(mouseX/100.0);
  shape(triShape);
  shape(quadShape);
  //shape(objShape);

  lights();
  println(frameRate);
}

PShape createShapeTri(PShape r) {
  PImage tex = loadImage("HeartC.JPG");
  PShape s = createShape();
  s.beginShape(TRIANGLES);
  s.noStroke();
  s.texture(tex);
  s.textureMode(NORMAL);
  for (int i=100; i&lt;r.getChildCount (); i++) {
    if (r.getChild(i).getVertexCount() ==3) {
      for (int j=0; j&lt;r.getChild (i).getVertexCount(); j++) {
        PVector p = r.getChild(i).getVertex(j);
        PVector n = r.getChild(i).getNormal(j);
        float u = r.getChild(i).getTextureU(j);
        float v = r.getChild(i).getTextureV(j);
        s.normal(n.x, n.y, n.z);
        s.vertex(p.x, p.y, p.z, u, v);
      }
    }
  }
  s.endShape();
  return s;
}

PShape createShapeQuad(PShape r) {
  PImage tex = loadImage("HeartC.JPG");
  PShape s = createShape();
  s.beginShape(QUADS);
  s.noStroke();
  s.texture(tex);
  s.textureMode(NORMAL);
  for (int i=100; i&lt;r.getChildCount (); i++) {
    if (r.getChild(i).getVertexCount() ==4) {
      for (int j=0; j&lt;r.getChild (i).getVertexCount(); j++) {
        PVector p = r.getChild(i).getVertex(j);
        PVector n = r.getChild(i).getNormal(j);
        float u = r.getChild(i).getTextureU(j);
        float v = r.getChild(i).getTextureV(j);
        s.normal(n.x, n.y, n.z);
        s.vertex(p.x, p.y, p.z, u, v);
      }
    }
  }
  s.endShape();
  return s;
}

DNA Lamp

2014-11-23T00:00:00Z

A prototype of a stolen idea from a friend of DNA sequence displaying LED lamp.

http://www.youtube.com/tCwi4XRZ3RU

My first with controlling addressable LEDs with Processing and the fantastic Fadecandy board from Micah Scott. Of course it would be way more fitting to have two LED strips and arrange them in a double helix...

Visualising SIFT descriptors

2014-11-02T00:00:00Z

Having worked on a few computer vision projects at work recently I've been interested in trying to understand what the computer is seeing. A lot of image processing algorithms involve putting the image through filters, or transforms, or extracting local descriptions of portions of the image. Often these are modelled around what the human vision system is understood to be doing, but of course our brain hides this low-levelling processing from us. It seems really interesting to try and visualise some of the intermediate steps of these algorithms to try and get a better understanding of how the computer is interpreting an image.

Of course it's nothing new, computer vision papers explaining or describing algorithms often tend to contain interesting visualisations (just try searching for eigenfaces or HOG pedestrian detection). I also came across a few really nice examples of people specifically trying to visualise (or reconstruct from) image features including: visualisation of HOG features, and image reconstruction from SIFT descriptors, both produce some fantastic and interesting images.

Visualising SIFT

The Scale Invariant Feature Transform is a commonly used method for detecting and describing local 'features' in an image, for a good description of what it is and how it works see the VLFeat API documentation. Basically SIFT produces features in the image that are local points of likely interest and distinctiveness, these are described by descriptors which take a small patch of pixels and compute local histograms of intensity gradients. I've used these descriptors several times for detecting and matching objects in scenes before, and have always wanted to better understand what the computer is seeing, and what it's giving importance to. Typically SIFT descriptors can be visualised as boxes with many arrows, which do give a hint of what the underlying algorithm is producing, but I wanted to try and produce something a little more visually pleasing (if less accurate).

I came up with a simple visualisation model for a SIFT descriptor. The descriptor is a 128 element vector representing the bins of a histogram of the local intensity gradients in an image over 16 patches around the keypoint. An example of my representation for a few keypoints is shown below:

http://youtu.be/RWi813cNN_k

Using OpenCV (in Python) to do the SIFT detection ^[1] 1. There is a good explanation of the slightly confusing OpenCV code for the SIFT keypoints here. and descriptions I placed descriptor visualisations into a blank image (scaled and rotated appropriately) and slowly saw some of the original structure of the image reappear in a ghostly form.

The colour in the images comes from doing feature detection in the red, green, and blue channels and adding appropriately coloured keypoints to the image. It's surprising just how much of the original details of the image begins to reappear, and it's also interesting to see what the SIFT algorithm pays attention to in the image and what it doesn't, (for example it has no interest in EVE in the frame from Wall-E above, she's just too sleek, uniform and smooth I guess).

The current algorithm is written in Python, and is painfully slow, so rendering short frames of video is fun, but takes a long time (I guess I need to look into how to use Cython, or learn C++...). The shimmering ghostly movement of rendered motion is particularly nice (and would be better if I could render it at higher resolution!)

http://youtu.be/Nw-8KRUAJBg

Another Wall-E image (in case you couldn't tell)

Abstract landscapes in Blender

2014-08-25T00:00:00Z

I recently saw some great work by Lee Riggs, using Maya to create wonderful abstract landscapes out of millions of small coloured block or pillars. They're fantastic images, and I wanted to try and emulate the effect in Blender as part of my ongoing efforts to try and get to grips with the program. It turns out it was a lot easier than I expected, just using a couple of modifiers and some duplication you can create some fairly nice effects. So here is a tutorial on how to achieve that blocky abstract landscape look with just an interesting image and some modifiers. Although it wasn't hard to get the basics in place, they're still not a patch on the originals, but the next plan is to generate some interesting images in Processing and use these as the basis for a landscape.

Blender abstract landscape tutorial

1. Find an image

First off find an image to turn into a landscape. Something colourful (or not?) and with some interesting structures seem to work well. I chose this stunning microscope image from Eckhard Völcker, any of his beautiful images would be fantastic, (his website and Flickr stream are highly recommended).

Load the image into your favourite image editing program and resize or crop the image to be square (it makes scaling the image correctly later a little easier). Save a colour copy, and save one in black and white (you might want to adjust the black and white conversion to try and get the height map you want), bear in mind that light areas will appear higher than dark.

The height map generated from the original image, white is higher.

2. Add some shapes

Fire up Blender and delete the starting cube. Add a plane, set the scale and dimension to 1 in both the x and y dimensions. Press tab to enter edit mode, and subdivide the plane 10 times.

Next, add your base object, it can be anything, but a cylinder or cube is a good starting point. Generally it’s going to have to be long and thin, but the exact dimensions will depend on how many subdivisions you’re going to apply to the parent object later.

Set up the dupliverts, as explained here. Basically, set the parent (Ctrl-P) of the object you created above, to be the plane, then go into the object menu and set duplication to vertices. So we have a grid of object that are going to make up our landscape, now we need a hell of a lot more and them, and also some landscape!

3. Add modifiers

We’ll add two modifiers, a displace modifier to create the landscape and a subsurf modifier to increase the number of vertices, and therefore landscape elements, we have in our scene.

First, add a simple subsurf modifier to increase the number of landscape elements. I've been using a number of subdivisions anywhere from 4 – 7 (7 of which gives you well over a million landscape elements, so be prepared for a bit of a slowdown when trying to find a good view in the scene). You’ll also probably want to adjust the x and y dimensions of the landscape elements to accommodate the changes in number caused by the different subdivision levels.

Next, add a displace modifier, and set the displacement texture to be the grayscale height map you created earlier. You’ll have to tune the strength of the displacement to give you the effect you want, but that’s a little tricky to tell until you have the elements set up. In general though a strength of 1 will be too much.

4. Set up the material

Set up the material of the landscape elements by selecting the original element and assigning a material something like the one illustrated in the image below.

Basically we are using the colour image as a texture input to the colour of the element, using the object info node to give the position of the element on the image. You’ll also have to add 0.5 to both the x and y values of the object position vector to make sure the origin of the image isn't in the middle of your plane.

5. Render!

Now you’re all set up, find a good view point, adjust the materials, lighting set-up, and geometry of the landscape elements and let Blender render the scene (I've been using Cycle rather than Blender internal). Some more example images I've generated are below.

Update: I've uploaded one of my .blend file that you can download here. I've been really happy to see people posting some examples they've made following this tutorial: 1, 2, 3, 4, and possibly 5, let me know if you make any more!

Water drop photography

2012-11-13T00:00:00Z

testing

After seeing many amazing water drop photos all around the internet I tried out some quick attempts at droplet collision photos. Armed with a cheap speedlite, a tray of water, and a syringe it was actually fairly easy to get some decent splashes. It still takes a lot of lucky timing, and many failed attempts, so building an arduino controlled water dropper sounds like a pretty fun project and evidently leads to some amazing pictures. There's also an amazingly large difference between milk and water at these scales, but the shapes both make are incredible.

Hovering

Pink water drop

A hand?

Milk and water drop ufo

Water splash crown

Water splash mushroom

Another mushroom

Justin Pinkney

Trailer Faces HQ Dataset

A dataset of 187 thousand high resolution face images from movie trailers! Download it from huggingface. #

Finding emotions #

TFHQ dataset #

Face identification #

Usage #

Experiments with Swapping Autoencoder

Spatially blending landscapes #

PCA latent editing #

CLIP guidance #

Unsynchronised style/content video generation #

Recursive content transformation videos #

Train rides #

The end #

Image Mixer Diffusion

Face Mixer Diffusion

Feed forward Dreambooth? #

Results - Making faces #

Back to CLIP #

Trailer Faces HQ and retrieval training #

Controlling face and background #

Where did I put that checkpoint again? #

Some related work #

Würstchen v2 Pokémon

Side notes and a Gallery

Sidenotes #

Gallery page #

Searching for Generative Train Journeys

The StyleGAN era #

Trying to move forward (pixel2style2pixel) #

Swapping autoencoder #

Next frame prediction with Stable Diffusion #

The final cut #

Stable Diffusion Image Variations

Results #

Version 2 #

Quirks (a.k.a the problem is always image resizing) #

Follow ups #

The Other Web

The other web #

The other web #

Links #

Algorithmic Film Making

Week 1 - Randomising clips #

Flowers Deconstructed

Experiments in Image Variation

Image inversion #

Latent editing with image variations

Text to Pokemon Generator

Catastrophic forgetting? #

More experiments #

Vivillon is the Rick Astley of Pokemon #

Coverage #

Style Space Face Editing

Pixel2Style2Pixel #

Style Space Analysis #

Other code #

Let's edit! #

Could try harder #

Blending the composite image #

Poor embedding #

Batching #

References #

Ukiyo-e faces dataset

Updates #

The dataset #

Further details #

Other datasets #

License and usage #

Making Toonify Yourself

Making an efficient Toonification model #

Deploying the model #

Toonification in the Wild #

The numbers #

Feedback #

The future #

Twitter #

Coverage #

Toonify yourself