When I got to the Recurse Center I wanted to write a lot of Rust, and get back into frontend development. I found a good balance between the two by writing an SPA with a background Web Worker in Rust.
I had the idea of making an automatic Paint-By-Number app, where you upload a picture and the app would turn it into one of these:
I didn’t know if this app existed before starting, so I googled around, and found ~2. One of them was the professional PBNify.com by Dan Munro.
It had source code available, and used a Web Worker to do the processing. This was perfect to build off of, because it meant I could translate just the Web worker and keep the rest of the app intact. I ended up fully rewriting it later, but I was able to reuse a lot of Dan’s code, and invaluable UX and CSS.
Interesting bits
- Setting up Webpack
- Rust->JS interop
- Using Svelte
- Performance
- Adding automatic color picking
- Deploying
Setting up Webpack
I actually spent my first week at RC just setting up the build pipeline for this! It was confusing for a couple of reasons.
The first is that there are several Rust+Wasm templates in the wasm-pack docs, and they all sound similar.
For my project, it sounded like I wanted “hybrid webpack integration”, but the template didn’t work well for me (kept on giving me wasm module loading errors)? It was confusing.
What really helped first was doing the Game of Life example in the Rust+Wasm book, and that used the other template (wasm-pack-template).
In the end I used a different template, the wasm-worker-template, which puts your Rust in a web worker, and merged it with the Svelte Webpack template.
It took me a whole day to merge them. They worked individually, but had a dependencies that differed in the patch version (the 0.0.1s), which I brushed off for a long time. It turned out a regression in Svelte had broken webpack interop, and what I had to do was bump the patch version numbers. This took me a day because it seemed so unlikely, I tried everything else first.
Rust | JS interop
Then it was off to the races to rewrite the Web Worker in Rust! I was a bit eager and set myself up to have to rewrite a lot of functions before I could see a result. It worked out this time and I got a matching result, but it was a bit risky. I got it to do the task in Rust!
My next point of interest/friction was passing arguments and return values between Rust and JS.
The function I was re-writing took in an image as a Bytebuffer and a color palette object.
The byte buffer was easy to describe as &[u8] in Rust, but the color
palette was an array of {r: , g:, b: } objects, which wasn’t as easy to input into the new #[wasm_pack] function.
In order to get something going quickly , I ended up turning that array into a byte buffer and then deserializing it by hand on the other side. Eh, it worked.
Later I had the same data interop problem in the other direction, when I wanted to return more than one image from my function.
In Rust I could send a struct back with both images, or a Vec of them.
But with Wasm, one is only allowed to return a type that implements the IntoWasmAbi trait (conversely, your function parameters have to have FromWasmAbi). There is an open issue to allow to return Vec<T> where T is IntoWasmAbi, and I think that will help ergonomics a lot.
There is hypothetically a way to return Rust defined structs and use them on the JS side, but I couldn’t get it to work.
I ended up putting the Rust side of the Web Worker in charge of the communication and using web-sys to call postMessage (the function that communicates with the main thread) in Rust, passing it a serde-json serialized struct. This works fine, so I can have Rust send status messages and data back as soon as it is computed.
Rewriting in Svelte
PBNify was written in an earlier style of javascript (it didn’t even have a package.json, how nice!), and it used Angular 1 in ways I didn’t understand/want to learn.
Part of my goal for this application was to get back into front-end development so I kept the CSS and the workflow, and re-wrote the front-end of the app (first in Vanilla JS). When the state management got too complicated for me (pretty quickly) I used Svelte. The point at which it got unwieldly was when I was trying to display lots of buttons for the UI and keep track of which button was visible depending on the app state. That’s when having a reactive library like Svelte came in handy.
Dan having thought out the styling and UX helped me a lot in this project.
Performance
In this project I wasn’t interested in clearing any particular performance bar, just that it would be fast enough, but I assumed WASM would be faster and was curious to see by how much more.
A fun (in hindsight) bit is that I got tripped up for half a day because my npm build script wasn’t setting cargo to release mode like I thoguht it was. So I initially thought WASM was twice as slow as JS, but when I got release mode into cargo, it turned out 10x as fast. In fact, the app is basically instantaneous, with nearly exactly the same implementation.
And since the old app usually took ~5-8 seconds per image, I consider this a usability boost, and something that actually helps the user instead of just making me as a programmer feel good that I re-wrote it in Rust.
I did tweak one algorithm: Turns out the slowest part of the whole process is putting the numbers in their “paint areas” or regions. You can’t just pick the center of the region because it might be oddly shaped and the center would not be in the shape, or would be in a weird place for the shape of the region. The old app had this slow method that with 3 or 4-ish nested for loops, where each pixel was considered, based on its distance from each edge, and each pixel took 2 nested for loops to find that distance.
I think I first tried to speed up the function by re-using some intermediate results, but it was complicated. The answer I picked was to first try the center of the region; if that is in the region, use that, otherwise fall back to the complicated algorithm which gives the nicer answer. This works fine.
The added performance headroom and Rust’s speed on this gave me confidence to implement a new feature: automatic color picking.
Auto color picking with K-Means
In the original app, after uploading an image, you had to choose which colors you wanted to Paint with. This is time consuming, it would be nice to have the computer do it for us.
My first hunch to try was K-Means clustering.
I will charitably attribute this to the magic of Rust, but my pairing partner & I were able to
add this in an hour and a half! We stole the following k-means code from Stackoverflow, modified Points to be
3d instead of 2d, and were done, we had auto color picking.
The code we copied ran 100 iterations, but we can only do 12 as wasm is slower.
In the future I would like to run the code for as many iterations as possible in <500ms.
Deploying
I’m always looking to level up my deploying&integration skills, because I believe that as you get better as a developer those skills become more and more important.
This time, deploying was straightforward and fun! One of my favorite aspect of Rust+WASM+JS projects is how clean the deploy ends up being.
I made a Dockerfile which combines the base Nodejs and Rust docker images. Then the Gitlab CI script uses that dockerfile, npm installs our packages, and runs npm run build. The rest is standard Gitlab Pages.
The whole thing is very little config and works flawlessly. The build takes ~5minutes, but that is completely great for the $0 I pay to get a Rust compilation on git push & static site hosting.
Pairing Highlights
There were many features I thought would be big and scary, but my pairing partners were amazing and helped me do them in < an hour and half!
- Lili helped me understand the code of PBNify, which set me up for success for the whole project
- Michael helped me add save to pdf! I was really afraid of this one, but Michael was so good at reading and implementing the spec, it was done in 1.5hr!
- George helped me read the EXIF data in the uploaded image to auto-rotate it, which was so cool to implement (though I didn’t finish putting it in :( )
- Ifema helped me add the K-means clustering in under an hour! I never thought it could be that easy.
- Harry helped me split the Image Loader into its own component and explained to me how to do state management in Svelte, which was invaluable.
- Johann walked with me through adding the little circle that highlight the colors you pick
- Miles helped me make the color distance a perceptual color distance, by using this approximation instead of pulling in the whole Lab crate
Conclusion
This actually ended up being my favorite app of RC. It was the easiest to pair on. It had lots of fun features to add, and was fun to play with for others.
It was the least daunting, I wasn’t trying to re-write a big system like Git or an OS or a distributed database to learn how it worked. It helped me keep moving every day and write code every day. I still found places to put algorithms and speed but I got a much more balanced experience and got to practice and learn a wider range of skills.
I wasn’t making it to feel smart or to have the “best” of anything, and it was still fun to play with the whole time. It really helped me re-orient the way I think about side-projects and why I code, which should be for fun, not to feel smart or to feel better than others.
I also really like that I have a laundry list of small, manageable bugs to fix on it, that I can use everyday as a warm-up for more self-directed learning learning.
The app is hosted here, try it out!
comments powered by Disqus