A Haskell Story: A Simple Image Hoster

Evaluating the Tools

Like a lot of "modern" programming languages, Haskell comes with the de-facto standard build system, in the case of Haskell cabal. I decided to use stack on top of that. For a small application like this, it was probably overkill and not necessary — however, I did want to play around with it, as it seems like a good choice for bigger applications.

For the web framework, Haskell has Yesod to offer. I did decide against using Yesod, as it seemed like way too much for such a simple application. The initial concept consisted of a single-file CGI script, and even after choosing Haskell to implement it, I did not want to "bring out the big guns". Instead, I used the cgi library, which looked decent and like the most fitting equivalent to Python's cgi module, which is what I would've used otherwise.

I was also looking at some templating engines to "spice up" the index page a bit with dynamically added content. Coming from Python I am a big fan of Jinja2, and I like the syntax and capabilities that it provides. Haskell has Ginger, which is inspired by Jinja and aims to provide the same features, so it seemed like a good option. I also glanced at Heist after stumbling over a reddit post, but preferred the Jinja syntax over more XML. Eventually though I did not end up using a template engine for the first version, simply because there was no need for it — the index page is currently completely static.

Besides that, there was not a lot to choose. I went with JSON for some configuration and metadata and used aeson to parse it, after I found a nice tutorial on how to use it. There was the alternative of json, but I do not know enough about the Haskell ecosystem to tell you which one is better, so I just went with the popular choice.

The Code

Writing the code itself turned out to be slightly better than anticipated. I do know about some functional programming concepts from programming in Scala and Rust, but the jump to a fully functional language like Haskell is still daunting. The good thing is that a lot of the script's functionality is pretty boring — take some data and save it into a file — so there weren't too many issues. My "monad stack" has CGI, IO and a global application settings object in it, and that was enough to do the job.

Working with aeson turned out to be nice as well. I am used to Rust's serde, and seeing that aeson can also derive FromJSON/ToJSON the same way that serde can derive Serialize/Deserialize is very useful and it prevents you from having to write a lot of boilerplate. But not just the JSON part worked nicely, the cgi module also looked like it was easier to use than expected. There was a bit of initial friction to get the monad stacking/lifting working properly, but after I switched from transformers to mtl (as that's what cgi seems to use), this got resolved rather quickly.

One of the more unusual bits is the amount of imports that I seem to need. The source file starts with

import Data.ByteString.Char8 (pack)
import Control.Applicative
import Control.Monad
import Control.Monad.Reader hiding (ask, asks)
import Crypto.BCrypt
import Data.Aeson
import Data.Aeson.Types
import Data.FileEmbed
import Data.Maybe
import Data.Time.Clock.System
import Data.Traversable
import GHC.Generics
import Network.CGI
import System.Directory
import System.FilePath
import System.Random

import qualified Control.Monad.Reader
import qualified Data.ByteString.Base64.Lazy as B64
import qualified Data.ByteString.Lazy as BS
import qualified Data.HashMap.Strict as HM
import qualified Data.Text as T

That seems like a lot — in Python, the following would probably offer around the same functionality:

import bcrypt
import json
import datetime
import cgi
import os
import random

This is not a criticism of Haskell (the number of imports seems like a silly metric to use), but more of a curiosity I found when developing. The Python standard library really is batteries included, none of the listed imports are a 3rd party module, and they barely scratch the surface of what Python provides out of the box.

In my opinion, the most awkward thing is the "check the prerequisites and then do stuff"-type of function. This happens when you check a bunch of stuff in sequence and then return early or call out to a different function to handle the request. For example, for the image upload, the script needs to get the user and password from the submitted form, check if the password matches and then make sure that the file content is not empty. In pseudo-Python, this could look like the following:

def handle_upload(form, settings):
    if form["user"] is None or form["password"] is None:
        return status(401)

    correct_password = settings.users.get(form["user"])
    if correct_password is None or form["password"] != correct_password:
        return status(401)

    if form["data"] is None:
        return status(400)

    # Proper code to upload here
    save_file_at(form["data"], random_name())

Of course, this is not perfect, especially since forgetting a None check could lead to crashes or have dangerous consequences. But the reason why I like this style is because it makes it clear that the authorization requirements are checked on top, and the main logic (if the authorization was successful) is still on a very low nesting level, making it nicer to read. Rust can do even better with the ? operator, providing type safety while also allowing for ergonomic handling of errors and early returns.

In Haskell, the Maybe and Either monads should be used for this type of thing. However, my code ended up looking like this:

handleUpload :: App CGIResult
handleUpload = do
    users <- asks users
    outdir <- asks outputDir
    user <- getInput "username"
    password <- getInput "password"
    let authorized = checkAuthorization users user password
    if authorized then do
        filecontent <- getImageData
        filename <- getInputFilename "imagefile"
        duration :: Maybe Integer <- readInput "duration"
        let finfo = (,,) <$> filecontent <*> (filename <|> Just "") <*> duration
        case finfo of
          Just (content, name, dur) -> do
              savedFileName <- saveFile content (takeExtension name) dur (fromJust user)
              redirect $ outdir </> savedFileName
          Nothing -> do
              setStatus 400 "Missing data"
              output "Invalid request"
     else do
         setStatus 401 "Unauthorized"
         output "Invalid credentials"

I am sure there are a lot of things wrong with this snippet and I would refrain from calling this good Haskell code, but you can see how the function drifts to the right. You can also clearly see that I am doing some manual error propagation by pattern matching — something that should better be handled in different ways. As a beginner though, this was the easiest way to write this function, without trying to get even more monads into my stack or fight with even more lift-s. In hindsight, this function should probably be rewritten in some way or another — even if it's just splitting the functionality into smaller functions. You can also see the fromJust, which is rather ugly (like unwrap() in Rust). It won't crash because checkAuthorization ensures that the user is not Nothing, but it should definitely be encoded in a different way (make checkAuthorization return a Maybe String? Or pass a closure to checkAuthorization that will only be called if the user is authorized?).

A similar drift can be seen in imgHostMain:

imgHostMain :: App CGIResult
imgHostMain = do
    cleanup <- getInput "cleanup"
    case cleanup of
      Just _ -> handleCleanup
      Nothing -> do
          method <- requestMethod
          case method of
            "POST" -> handleUpload
            _ -> output indexPage

Here the problem is not prerequisite checking, but just the fact that depending on the circumstances, a different function has to be called. Again, in the Python version, this could probably look more like this:

def img_host_main(form):
    if form["cleanup"]:
        return handle_cleanup()
    if form.method == "POST":
        return handle_upload()

    return show_index()

This variant clearly shows show_index() as the "default" path, and the other ones as special paths depending on what the script is currently doing. Again, I would prefer the Python version, but did not yet find a nice way to achieve a similarly pleasant way to handle this in Haskell.

A cool thing that I like is embedFile, which is like Rust's include_bytes!() macro. It's useful to include the small index page directly into the executable, without the need to fiddle with file-loading logic at runtime:

indexPage :: String
indexPage = $(embedStringFile "src/index.html")

Here, embedStringFile is like include_str!() in Rust and provides the file contents as a textual string (as opposed to the raw bytes).

All in all, the code is pretty short though: The file has 302 lines, including blanks, Haddock comments and imports/language extensions. cloc reports 187 lines of actual code, which is probably still more than a compact Python implementation, but using Python is cheating.

The Woes of File Uploads

Given that cgi advertises "support for file uploads", I did not expect this part to become a problem. However, I quickly realized that the code I had written to handle a file upload did not work — at all. It appeared as if no file was ever there, not even the filename could be extracted. And that is despite the file upload being as simple and easy as writing fileContent <- getInputFPS "form-field-name".

A bit let down I set out to find the source of my problem. Of course, being rather inexperienced in the language I'm working with did not speed up the process. However, I found the Practical web programming in Haskell tutorial in the official Haskell wiki, and it has a part on file uploads! I compared their code to mine, only to realize that I couldn't find a difference — but surely, the code in the wiki must be working! So I quickly replaced my whole Main.hs with their code, and lo and behold... it was still broken.

The search for the bug continued. I wrote a small Python script to ensure that the Python http.server that I use for development can actually handle file uploads, and it worked. I continued playing around with Haskell's cgi until I stumbled upon issue #47 on their GitHub, noting that this seems to be a (more than a year old) bug in cgi. I downloaded the code for the library and played around with it, until I arrived at parseMultipartBody, a function defined in multipart, which seemed to be the culprit. This lead me to issue #4 from last September, having some more information in it.

Downloading multipart and reverting the commits mentioned in the bug report made file uploads work again!

All in all, this bug took me longer to track down than I would like to admit. Going to cgi's GitHub directly and checking the issues would've saved me some time and headache, however I did not expect the issue to lie in the library. I was first blaming my code and my (development) setup, and only when it became clear that there was no issue, I considered that it might be a bug in cgi. This is also due to cgi being a relatively old, battle-tested and stable library, which I did not expect to break for such a rather basic functionality. Then again, the demand for old-fashioned CGI scripts written in Haskell is probably rather low, allowing for that bug to go unnoticed and unfixed for longer time.

At some point along the process I also landed on the old repository for cgi, which hasn't been maintained in 6 years — and which doesn't have the important issue. This is to be blamed completely on me though, I did not check the GitHub user and should've used the repository link in Hackage straight away.

A Song of Static Linkage

With the initial (Python) concept, the idea was to have a single file without external requirements that you could just drop wherever you wanted your image hosting script to be. In the case of Python, this works because Python comes pre-installed with all necessary modules in the standard library. For Haskell, I decided that I want to statically link my binary, which means that the resulting executable will contain a copy of every library in use. I'm not here to discuss the (dis)advantages of static linking versus dynamic linking, but I decided that I am fine with the disadvantages as long as it makes the deployment easier for me.

I did find some information about how to achieve that in Haskell, which mostly consisted of adding -static and similar flags to the ghc-options. That alone didn't work however, it bombarded me with hundreds of errors about code needing to be compiled with -fPIC. I also found Static compilation with Stack, a blog post on FPComplete. That scared me off a bit, I didn't expect that I would need Docker and a hack modifying some system libraries just to get a static binary:

WORKDIR /usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/
RUN cp crtbeginT.o crtbeginT.o.orig
RUN cp crtbeginS.o crtbeginT.o

This turned out to be unnecessary in the end, so that is a good thing. The final clue was found in a reddit post, asking for an "easy way to compile static binaries". Adding

ghc-options: -Wall -O2 -static -threaded
cc-options: -static
ld-options: -static -pthread
extra-lib-dirs: ./.stack-work/lib

to my package.yml was already most of the work. Then I downloadad gmp from https://gmplib.org, compiled it manually and copied the static library (.libs/libgmp.a in the gmp build folder) into the .stack-work/lib folder. I have a feeling that this should be automated in a better way as well — the AUR does provide a libgmp-static package, but it has been flagged as out of date. Otherwise, the build process in Setup.hs could be modified to automatically build libgmp.a if not available, but that opens up another can of worms that I was not willing to deal with yet.

Conclusion

Use the right tool for the job. In my case, sticking to the original Python idea would have probably lead to a faster result and cleaner code, however, I did not want to pass up the opportunity to get some more Haskell exposure.

The code was not too hard to get working, although somtimes it felt a bit like "trying to add a lift and see if that fixes the problem" instead of understanding what is really going on. That seems to be normal when entering a new world of concepts though, and I assume that it will get better when working more with "real life" Haskell outside of simple monad tutorials.

I am not 100% happy with how the code looks. The aforementioned drift problem and the symbol soup at times look a bit weird and I'm sure that there are a lot of improvements to be done there. It feels like using a for i in range(len(xs)) in Python, where understanding of what is going on is superficial and the concepts of for-loops and iterators are not engrained in one's brain yet. But everyone starts small, and it surely was exciting to get it working in a decent time!

The final code is available on GitHub and this post is referencing commit 63b2ff187e290d8eafd39a902af56a88c6ab53e9 (Initial commit).