Remember the two benefits of failure. First, if you do fail, you learn what doesn’t work; and second, the failure gives you the opportunity to try a new approach 

Roger Von Oech

Be Transparent

One of the most important things you can do when you screw up is to be transparent. Let your users know what happened, why it happened, what you’re doing to fix it and how you’re going to prevent it from happening in the future. While this doesn’t fully restore your user’s faith in your development team and the product it goes a long way to repairing the broken bridges.

Constant updates help your users know the progress that you’re making to recover from failure and can give them an idea of when everything will be back to normal (if the problem is more than a simple fix).

Apologize

You made the mistake, you should apologize. If you broke someone’s phone you would apologize and this is no different. By apologizing you’re admitting that you’ve done something wrong and are sorry about breaking something that people use. Apologizing isn’t hard, a simple sorry is enough, but make sure you actually apologize and not just pseudo-apologize.

Fix it Quickly

This one is pretty straightforward. The longer an issue exists the more likely you’re going to lose customers. If the problem exists for more than a couple of hours you should look into compensating your users with either a discount or something else of value. This, in combination with an apology and transparency, should help you keep your users through your failures.

Learn from it

The best thing you can do after failing is to learn from it.

The only real mistake is the one from which we learn nothing

Henry Ford

This is where postmortems come in. By documenting your failures you make it easier to learn from and easier to prevent in the future. You’re building out a repository of information that new developers can lean on when deploying new code. This also helps you prevent similar issues from happening in the future.

The post Properly Handling Failures appeared first on Procrastinating Developer.

TL;DR

I created a computer program that generates a unique lightsaber made out of four different parts (blade, hilt, pommel and button) and tweets it out once a day along with some statistics about the lightsaber.

Getting Started

The overall premise of this script is fairly simple. It randomly selects four pieces of a lightsaber and then pieces them together. To achieve this I used an online tool called Piskel to generate all of the 8-bit parts. While Piskel is normally used to create animated sprites for websites it’s one of the better 8-bit art editors. My favourite feature from it is the ability to mirror lines. This simplifies the process of making symmetrical lightsabers.

Once I created a base set of parts put them in an images directory. This directory has 5 folders, one for each part and an output folder (lightsabers). This allows me to keep everything organized as well as helps later on when generating lightsabers.

images/
    blades/
        b1.png
        b2.png
    hilts/
        h1.png
        h2.png
    buttons/
        u1.png
    pommels/
        p1.png
    lightsabers/
        h1b2u1p1.png

To do all of this I installed three python packages. I use Pillow (version 7.1.1) for all of the image processing, Tweepy (version 3.8.0) to send out the tweets and numpy (version 1.18.3) to do some colour processing that I’ll discuss later on. Overall it’s a very simple requirements.txt

# requirements.txt
numpy==1.18.3
Pillow==7.1.1
tweepy==3.8.0

The first version of this script was really this simple. Call generate_lightsaber and that’s it. Up next I’ll talk about how I generated the initial lightsaber (just the blade and hilt), then how I added buttons and pommels and finally how I tweeted the final image.

# lightsaber.py
import os, random
from pathlib import Path, PurePath
# This is actually Pillow but it's API is backwards compatible with the older PIL
from PIL import Image

def generate_lightsaber():
    # Will talk about this more next

if __name__ == "__main__":
    generate_lightsaber()

Adding Blades and Hilts

In my initial design, I only had blades and hilts so that’s where I started. My first goal was to get a single blade and a single hilt merged and properly lined up on a single image. I first added code to fetch a unique lightsaber part based on the directory layout above.

# lightsaber.py
# Set some constants to keep the code clean and consistent
IMAGE_PATH = Path('../images')
BLADE_PATH = IMAGE_PATH / 'blades'
HILT_PATH = IMAGE_PATH / 'hilts'
OUTPUT_PATH = IMAGE_PATH / 'lightsabers'

def fetch_lightsaber_parts():
    hilt = Path(f"{HILT_PATH}/{random.choice(os.listdir(HILT_PATH))}")
    blade = Path(f"{BLADE_PATH}/{random.choice(os.listdir(BLADE_PATH))}")

    return (hilt, blade)

First, we have os.listdir(), which returns a list of filenames in the directory given. The list is in arbitrary order and it does not include the special entries ‘.’ and ‘..’ even if they are present in the directory. It does, however, include any folders or special files (like .DS_store in OS X) so make sure your folder only includes the image files. We pass that filename into random.choice which will choose a random filename. Finally, we create the full file path and return that for both the hilt and the blade.

Once we’ve gotten the image paths we can start piecing them together into a final image.

# lightsaber.py
def generate_lightsaber():
    blade_path, hilt_path = fetch_lightsaber_parts()

    # Open the images using Pillow
    blade = Image.open(blade_path, 'r')
    hilt = Image.open(hilt_path, 'r')

    # Open the output image. Twitter displays the entire image if it's 1024x512
    output = Image.new("RGB", (1024, 512), (255, 255, 255))

    # Paste the blade and hilt onto the output image. 
    output.paste(blade, blade_offset, mask=blade)
    output.paste(hilt, hilt_offset, mask=hilt)

    # Save the output image to disk
    img.save("{}/{}.png".format(OUTPUT_PATH, output_filename))

First, we’ll fetch the paths as described above. After that, we’ll use Pillow to open the images as well as create a new output image with the width and height specified by Twitter to allow us to show the entire image as a preview. Finally we paste the two original images and save the output.

When pasting the images onto the output you’ll see two arguments that I haven’t shown yet, <part>_offset and mask. The offset is used to determine where the parts are placed on the output image. When you get this wrong, funny things can happen.

To calculate the offset, where the image gets placed, I need to get and store all of the dimensions of the images. Then we calculate the middle position for the width (output_w - blade_w) // 2 this aligns it horizontally on the image. This is true for both the blade and the hilt.

To position the blade and hilt vertically it requires a little more math. For the hilt, we take the height of the output image (512px) and subtract the height of the hilt image (between 70 and 120px depending on the design). This makes the hilt start at the bottom of the image.

# lightsaber.py
output_w, output_h = output.size
blade_w, blade_h = blade.size
hilt_w, hilt_h = hilt.size

blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)
hilt_offset = (output_h - hilt_h, (output_w - hilt_w) // 2)

Using the array above as an example, if we had a lightsaber 1px wide and 2px (1px hilt, 1px blade) tall the math would be.

blade_offset = ((4 - 1 - 1), (5 - 1) // 2 ) # (2, 2)
hilt_offset = ((4 - 1), (5 - 1) // 2) # (3, 2)

Finally, we have to pass in the original image as a mask, otherwise, Pillow will default the transparent backgrounds into black. By using a mask Pillow only writes data to the output image for the pixels that have colour. The image below on the left has a mask applied, the one on the right does not.

Adding Hilt Offsets and Extra Information

When calculating the blade offset you may have noticed a variable hilt_offset. In some designs, the part where the blade comes out of the hilt isn’t the start of the lightsaber. In these cases I needed the blade to start “below” where the hilt starts. Knowing that I needed to do this but also store metadata on hilts, blades, buttons and pommels down the road I created a manifest.py file to store these defaults.

# manifest.py
MANIFEST = {
    "hilt": {
        "h1": {
            "offsets": {
                "blade": 5
            }
        }
    }
}

# lightsaber.py
from manifest import MANIFEST

def get_hilt_offset(hilt):
    return MANIFEST['hilt'][hilt]['offsets']['blade']

def generate_lightsaber():
    ...
    hilt_offset = get_hilt_offset(hilt)
    blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)
    

Without this offset, I was originally getting a floating blade for certain designs. You can see below that the bottom of the blade lines up with the top of the rightmost side of the lightsaber but it still looks funny.

This manifest file also holds information on button offsets, colours and tweet information but I’ll go more in-depth into that later on.

Adding Buttons

After getting the blades and hilts lined up I realized that to hit 10,000 unique lightsabers with only blades and hilts I would need to design 100 of each (200 total). My 8-bit art skills just aren’t up to that. By adding buttons I could cut down the unique designs needed only 22 each (66 total). Buttons were added to the final image in exactly the same way as hilts and blades.

# lightsaber.py
# Set some constants to keep the code clean and consistent
IMAGE_PATH = Path('../images')
BLADE_PATH = IMAGE_PATH / 'blades'
HILT_PATH = IMAGE_PATH / 'hilts'
BUTTON_PATH = IMAGE_PATH / 'buttons'
OUTPUT_PATH = IMAGE_PATH / 'lightsabers'

def fetch_lightsaber_parts():
    hilt = Path(f"{HILT_PATH}/{random.choice(os.listdir(HILT_PATH))}")
    blade = Path(f"{BLADE_PATH}/{random.choice(os.listdir(BLADE_PATH))}")
    button = Path(f"{BUTTON_PATH}/{random.choice(os.listdir(BUTTON_PATH))}")

    return (hilt, blade, button)

def generate_lightsaber():
    blade_path, hilt_path, button_path = fetch_lightsaber_parts()

    # Open the images using Pillow
    blade = Image.open(blade_path, 'r')
    hilt = Image.open(hilt_path, 'r')
    button = Image.open(button_path, 'r')

    # Open the output image. Twitter displays the entire image if it's 1024x512
    output = Image.new("RGB", (1024, 512), (255, 255, 255))

    output_w, output_h = output.size
    blade_w, blade_h = blade.size
    hilt_w, hilt_h = hilt.size
    button_w, button_h = button.size

    hilt_offset = get_hilt_offset(hilt)
    blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)
    hilt_offset = (output_h - hilt_h, (output_w - hilt_w) // 2)

    # Paste the blade and hilt onto the output image. 
    output.paste(blade, blade_offset, mask=blade)
    output.paste(hilt, hilt_offset, mask=hilt)
    output.paste(button, get_button_offset(hilt), mask=button)

    # Save the output image to disk
    img.save("{}/{}.png".format(OUTPUT_PATH, output_filename))

The main complication with buttons is that they can be placed in a square on the hilt and this square is different for each hilt. There are a few different ways we could do this automatically (detect edges, detect width and height of the blade, etc…) but for simplicity’s sake, I just decided to manually calculate it for each hilt and store it in the manifest file.

# manifest.py
"hilt": {
    "h1": {
        "offsets": {
            "blade": 0,
            "button": {
                "x": (8, 9),
                "y": (110, 111)
            },
        },
    },
}

# lightsaber.py
def get_button_offset(hilt):
    between_x = MANIFEST['hilt'][hilt]['offsets']['button']['x']
    between_y = MANIFEST['hilt'][hilt]['offsets']['button']['y']

    return (random.randint(between_x[0], between_x[1]), random.randint(between_y[0], between_y[1]))

Adding Pommels

Once again, pommels were added to the image in much the same way as the others only this time I needed to update the height of both the blade and the hilt to accommodate the pommel. This didn’t go well at first.

# lightsaber.py
# Set some constants to keep the code clean and consistent
IMAGE_PATH = Path('../images')
BLADE_PATH = IMAGE_PATH / 'blades'
HILT_PATH = IMAGE_PATH / 'hilts'
BUTTON_PATH = IMAGE_PATH / 'buttons'
POMMEL_PATH = IMAGE_PATH / 'pommels'
OUTPUT_PATH = IMAGE_PATH / 'lightsabers'

def fetch_lightsaber_parts():
    hilt = Path(f"{HILT_PATH}/{random.choice(os.listdir(HILT_PATH))}")
    blade = Path(f"{BLADE_PATH}/{random.choice(os.listdir(BLADE_PATH))}")
    button = Path(f"{BUTTON_PATH}/{random.choice(os.listdir(BUTTON_PATH))}")
    pommel = Path(f"{POMMEL_PATH}/{random.choice(os.listdir(POMMEL_PATH))}")

    return (hilt, blade, button, pommel)

def generate_lightsaber():
    blade_path, hilt_path, button_path, pommel_path = fetch_lightsaber_parts()

    # Open the images using Pillow
    blade = Image.open(blade_path, 'r')
    hilt = Image.open(hilt_path, 'r')
    button = Image.open(button_path, 'r')
    pommel = Image.open(pommel_path, 'r')

    # Open the output image. Twitter displays the entire image if it's 1024x512
    output = Image.new("RGB", (1024, 512), (255, 255, 255))

    output_w, output_h = output.size
    blade_w, blade_h = blade.size
    hilt_w, hilt_h = hilt.size
    button_w, button_h = button.size
    pommel_w, pommel_h = pommel.size

    pommel_offset = pommel_h
    hilt_offset = get_hilt_offset(hilt_name) - pommel_offset
    button_offset = get_button_offset(hilt_name)  

    blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)
    hilt_offset = (output_h - hilt_h - pommel_offset, (output_w - hilt_w) // 2)
    pommel_offset = (output_h - pommel_h, (output_w - pommel_w) // 2)

    # Paste the blade and hilt onto the output image. 
    output.paste(blade, blade_offset, mask=blade)
    output.paste(hilt, hilt_offset, mask=hilt)
    output.paste(button, get_button_offset(hilt), mask=button)

    # Save the output image to disk
    img.save("{}/{}.png".format(OUTPUT_PATH, output_filename))

The next challenge I encountered with pommels was designing them so they looked nice across all hilts. When I first designed the lightsaber I included both the hilt and the pommel. When I realized I needed to split them out to allow for more unique combinations I simply cut the old 8-bit images in two. The problem with this is that the colour schemes between hilts and pommels just didn’t match.

After a lot of thinking I decided that I could do something similar to how green screens work. If I designed the images with specific colours (in this case Red = Primary, Blue = Secondary and Green = Tertiary) I could pull them out using Pillow and Numpy and substitute them for the colours I want.

# manifest.py
"hilt": {
    "h1": {
        "offsets": {
            "blade": 0,
            "button": {
                "x": (8, 9),
                "y": (110, 111)
            },
        },
        "colours": {
            "primary": (216,216,216), #d8d8d8
            "secondary": (141,141,141), #8d8d8d
            "tertiary": (180, 97, 19), #b46113
        },
    },
}

# lightsaber.py
import numpy as np

def convert_colours(img, hilt):
    img = img.convert('RGBA')
    data = np.array(img)

    # Grab the pixels which are 100% red, 100% blue and 100% green
    red, green, blue, alpha = data.T
    primary = (red == 255) & (blue == 0) & (green == 0)
    secondary = (red == 0) & (blue == 255) & (green == 0)
    tertiary = (red == 0) & (blue == 0) & (green == 255)

    # Substitute out the colours for the hilt colour scheme
    data[..., :-1][primary.T] = MANIFEST['hilt'][hilt_name]['colours']['primary']
    data[..., :-1][secondary.T] = MANIFEST['hilt'][hilt_name]['colours']['secondary']
    data[..., :-1][tertiary.T] = MANIFEST['hilt'][hilt_name]['colours']['tertiary']

    return Image.fromarray(data)

def generate_lightsaber():
    ... 
    pommel = Image.open(pommel_path, 'r')
    pommel = convert_colours(pommel, hilt_name)
    ...

Once I did that everything was all set. The pommels now have the correct colour scheme as the lightsaber.

Generating Random Tweet Text

The final part of all of this is generating the actual tweet. I added a number of fields to the manifest file including the hilt length and material, the blade colour, crystal and who used it and finally the pommel length. After that a new function, generate_tweet_text, that pulls all of this new information together and generates the text to tweet out.

# manifest.py
"hilt": {
    "h1": {
        "offsets": {
            "blade": 0,
            "button": {
                "x": (8, 9),
                "y": (110, 111)
            },
        },
        "colours": {
            "primary": (216,216,216), #d8d8d8
            "secondary": (141,141,141), #8d8d8d
            "tertiary": (180, 97, 19), #b46113
        },
        "length": 24,
        "materials": "Alloy metal/Salvaged materials"
    },
},
"blade": {
    "b1": {
        "colour": "Red",
        "crystal": ["Ilum crystal", "Ultima Pearl"],
        "type": "Sith"
    },
},
"pommel": {
    "p1": {
        "length": 5,
    },
}

# lightsaber.py
AVERAGE_HILT_LENGTH = 25
AVERAGE_POMMEL_LENGTH = 3
AVERAGE_BLADE_LENGTH = 90

NAMES = ['List', 'of', 'generated', 'names']

def generate_tweet_text(hilt, blade, pommel):
    hilt_details = MANIFEST['hilt'][hilt]
    blade_details = MANIFEST['blade'][blade]
    pommel_details = MANIFEST['pommel'][pommel]

    hilt_length = hilt_details['length']
    pommel_length = pommel_details['length']

    total_length = hilt_length + pommel_length
    average_length = AVERAGE_HILT_LENGTH + AVERAGE_POMMEL_LENGTH
    blade_length = int(AVERAGE_BLADE_LENGTH * (total_length / average_length))

    title = blade_details['type']
    if type(title) is list:
        title = random.choice(title)

    crystal = MANIFEST['blade'][blade]['crystal']
    if type(crystal) is list:
        crystal = random.choice(crystal)

    name = f"{title} {random.choice(NAMES)}"

    tweet = f'''Owner: {name}
Hilt Length: {total_length} cm
Blade Length: {blade_length} cm
Blade Colour: {MANIFEST['blade'][blade]['colour']}
Kyber Crystal: {crystal}

#StarWars
'''

    return tweet

To generate the unique names I used this Star Wars name generator to generate a number of names. I then randomly choose one to use.

To tweet out the final tweet I use the python library Tweepy to do all of the heavy lifting for me. Since I already have the image saved on file all I need to do is grab the credentials from an environment variable, upload the media and then post the tweet.

consumer_key = os.getenv('CONSUMER_KEY')
consumer_secret = os.getenv('CONSUMER_SECRET')

access_token = os.getenv('ACCESS_TOKEN')
access_token_secret = os.getenv('ACCESS_TOKEN_SECRET')

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)

tweet_text = generate_tweet_text(hilt, blade, pommel)

media = api.media_upload(path)
api.update_status(status=tweet_text, media_ids=[media.media_id,])

If this is your first time setting up a twitter bot you’ll need to create an application first, grab your credentials and then store them in an environment variable.

Wrapping it all up

Now that everything’s running correctly, I’ve got it generating a unique lightsaber and posting it to Twitter I just have to add it to a Cron to run every day.

12 21 * * * python3 lightsaber.py >> lightsaber.log

This will run the program, every day at 9:12 pm. And with that, we’re done! We’ve got a Python program that generates a unique lightsaber, every single day, and tweets it out. Which Jedi/Sith owns your favourite lightsaber?

The post Using Python to generate over 10,000 unique 8-bit lightsabers appeared first on Procrastinating Developer.

The main feature of Daily Lightsaber is the scheduled post of a new lightsaber every day. To do this I needed to add Cron to my droplet. This guide assumes you have a droplet running with Ubuntu 18.04.

Installing Cron

The first thing you’ll want to do is make sure your droplet’s package manager is updated to the latest version.

$ sudo apt update

Then you can simply install cron

$ sudo apt-get install cron

Editing your Crontab

Once you’ve installed cron onto your droplet you can edit your crontab and start adding entries to run automatically. A crontab is a special file that holds the schedule of jobs cron will run. To edit your crontab you can type:

$ crontab -e

The first time you run this command you’ll be given the option to choose a text editor to edit it with. If you don’t have a preference, nano is probably the most user friendly.

no crontab for <user_name> - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.basic
  3. /usr/bin/vim.tiny
  4. /bin/ed

Choose 1-4 [1]: 

Once you’ve chosen you’ll see the default crontab text. You can remove all of the comments if you’d like or simply add a new line at the bottom.

# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command

Each line in the crontab will execute a single command. Below I execute a python script everyday at 9:12pm and pipe any output to a log file.

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * command to execute
12 21 * * * python3 lightsaber.py >> lightsaber.log

Now that you have cron installed and you’ve edited your crontab you need to make sure it’s enabled. This is the final step to get everything working.

$ sudo systemctl enable cron

Now you have a fully functioning cron on your droplet. This will allow you to run commands at a set schedule. If you’re interested in getting set up with Digital Ocean you can use this link to get $100 in free credits.

The post Adding Cron to a Digital Ocean Droplet appeared first on Procrastinating Developer.

Problem Solving

For as long as I can remember I’ve loved solving puzzles. When I would get a new puzzle I’d try to break it down to smaller logical chunks to try to see how the puzzle worked. That’s exactly what I get to do at work every day. I first start with a large overarching problem that’s given to me by a customer and then I have to break it down into smaller logical chunks. Sometimes these problems aren’t exciting or challenging. I’ve written enough CRUD websites that they no longer give challenges but sometimes these problems are unique and fun.

Building Stuff

While I love solving problems and it’s fun to have a challenging career I really enjoy building things. As a kid I was always taking stuff apart, finding out how it worked and then building it back. Writing software provides me with quick gratification because I can see the immediate results of my work. I can quickly put together a new app and once deployed see it being used instantly on the Internet.

Being a developer also lets me build a large range of things. I can build a website one day, an iPhone app another and configure a server on the weekend. This diversity allows a developer to feel new and exciting even if you’ve done it for a few decades.

Continuous Learning

As a developer, if you’re not learning the latest technology you get left behind very quickly. There’s always a new language coming out or a new framework to read about. While this can become overwhelming at first I often find trying a new language out to be a fun experiment on a day off. I’ve also found that this constant learning makes programming stay fresh and fun.

I’ve since been promoted to Manager and now a Director but I think these concepts still ring true, just at a different scale. While I don’t directly build applications and websites anymore I get to build more things by building an amazing team. I’d love to hear why you became a developer. Let me know in the comments below.

The post Why I Became a Developer appeared first on Procrastinating Developer.

In this article, I’m going to outline how we reduced the file size of our backup that in turn sped up our restores even further.

Investigation into Size of Backup

When I first wrote about reducing the size of our backups the compressed backup size was sitting around 2GB (30GB uncompressed). Over time our database almost doubled (3.7GB compressed, 68GB uncompressed). This meant that not only was it taking significantly more time to restore, it was also taking double the time to transfer the file across.

With the knowledge that the backup had doubled in size, I started trying to figure out why and where the data was growing the most.

The first thing I did was actually find out what the uncompressed size of the database was.

# SELECT pg_size_pretty(pg_database_size('dbname'));

pg_size_pretty
----------------
 68 GB
(1 row)

Next I decided to see if I could dig into the sizes of the individual tables to see if there were any big culprits.

# SELECT table_name, pg_relation_size(table_name), 
pg_size_pretty(pg_relation_size(table_name))
FROM information_schema.tables
WHERE table_schema = 'public'
ORDER by 2 DESC;

table_name         | pg_relation_size | pg_size_pretty
-------------------+------------------+---------------
 logging           |      19740196864 | 18 GB
 reversions        |      15719358464 | 15 GB
 webhook_logging   |       8780668928 | 8374 MB
 ...               |       1994645504 | 1902 MB
 ...               |        371900416 | 355 MB
 ...               |        304226304 | 290 MB

As you can see the top 3 tables, which make up over 60% of the entire database, are all related to history or logging, both of which generally aren’t necessary when restoring your dev environment. When you include the indexes to those tables (17GB, 1GB, 1.5GB respectively) these tables make up 89% of the database. With this information in hand, I decided to move on from discovery (89% reduction is good enough for me) and see if I could exclude these tables from our dev backups.

Minimizing the Backup

Whenever I’m approaching a problem I like to start by reading the project documentation. PostgreSQL’s docs are fantastic and after a few minutes of reading through pg_dump‘s documentation I found exactly what I needed.

pg_dump dbname -Fc \
   --exclude-table-data 'logging*' \
   --exclude-table-data 'reversions*' \
   --exclude-table-data 'webhooks_logging*' > postgres.dev.sql

* is used as a wildcard to remove the indexes for these tables as well.

By specifying the --exclude-table-data parameter I was able to reduce our database size from 3.7GB compressed (68GB uncompressed) to GB compressed 0.7GB (5.4GB compressed).

As you can see below the results are pretty great. It sped up

Previously:

$ pg_restore -d db -j 8 dumpfc.gz
real 16m49.539s
user 1m1.344s
sys 0m39.522s

Now:

$ pg_restore -d db -j 8 devfc.gz
real 5m38.156s
user 0m24.574s
sys 0m13.325s

As you can see, removing those tables removed 89% from the total size and sped up the overall restore time by 66%! If you remember back to part 1 we started off with an initial restore time of 32.5 minutes. This means we’ve managed to improve the restore time by over 26.9 minutes or 87%.

In the end, our active restore time got cut from 16 minutes to 5 minutes. This saves us an additional 57 hours of restore time a year (6 devs, 52 restores a year, 11 minutes). In total, we’ve removed 130 hours worth of time waiting for restores to happen.

Final Tips and Thoughts

Going back to the PostgreSQL docs there’s a number of things we could do to potentially make restoring faster. Things such as using the -j parameter on pg_dump to make backing up our database quicker (only available on PostgreSQL 9.3+). Disabling autocommit , increasing the maintenance_work_mem value to be much larger or setting the max_wal_size value to be larger as well.

At this point, I’m happy with the overall time it takes to restore our local environments.

The post Speeding up Postgres Restores Part 2 appeared first on Procrastinating Developer.

Below is the process I took to speed up our DB restores.

Naive Approach

Below is essentially our first version of backing up and restore. On our main Postgres database, we’d run a simple pg_dump and pipe the output into gzip. When we wanted to restore our dev environment we’d SCP the compressed file over, uncompress it and then just load it via psql command.

$ pg_dump db | gzip > dump.gz
real 7m9.882s
user 5m7.383s
sys 2m56.495s

$ gunzip dump.gz
real 2m27.700s
user 1m28.146s
sys 0m41.451s

$ psql db < dump
real 30m4.237s
user 0m21.545s
sys 0m44.331s

Total time for naive approach: 39 minutes, 41 seconds (32.5 minutes restoring dev)

This worked quite well for a long time. It’s simple, easy to set up and was fast when our DB was only a few hundred MB. Obviously though, 32.5 minutes to restore your dev database is just unacceptable.

Pipe Uncompress

My initial idea to speed up our restore time was to simply pipe the compressed file directly into the psql command by using zcat. You can think of zcat as the same as the cat command but for compressed files. It uncompresses the file and prints it to the standard output which you can then pipe into your psql command.

$ pg_dump db | gzip > dump.gz
real 7m9.882s
user 5m7.383s
sys 2m56.495s

$ zcat dump.gz | psql db
real 26m22.356s
user 1m28.850s
sys 1m47.443s

Total time: 33 minutes, 31 seconds (26.3 minutes restoring dev, 20% faster)

Excellent, this sped up the process by 16% overall, 20% on the actual restore. Because I/O was a bottleneck, not writing to disk saved us over 6 minutes. Overall though I was still not happy. 26 minutes wasted on restoring our dev database wasn’t good enough, I had to go further.

Custom Format

As I dug into the pg_dump documentation I noticed that by default, pg_dump exports a plain-text SQL file. As you can see above, we then gzip’d it to make it smaller to store it. Postgres provides a Custom format that, by default, uses the zlib to compress it. My initial thinking was that if Postgres is already writing the file to disk in the plain-text format, it’d be faster to have Postgres also compress it at the same time instead of having to pipe it to gzip.

Because of this custom format, I had to switch to using pg_restore since you can’t redirect the compressed file using psql.

$ pg_dump -Fc db > dumpfc.gz
real 6m28.497s
user 5m2.275s
sys 1m16.637s

$ pg_restore -d db dumpfc.gz
real 26m26.511s
user 0m56.824s
sys 0m15.037s

Total time 32 minutes, 54 seconds (26.4 minutes restoring dev).

I was right in thinking that the actual backup process would be faster since we didn’t have to pipe the output to gzip. Unfortunately, restoring this custom format on your local machine didn’t result in the process being any faster. Back to the drawing board.

Specifying Jobs

The first thing I always do when I’m diving into a problem is to read the documentation and source code. Postgres has fantastic documentation with clearly laid out and labelled options. One of these options is the ability to specify the number of jobs that run concurrently while pg_restore is doing the most time-consuming parts, load data, create indexes, or create constraints.

pg_restore docs say that a good place to start with the number of concurrent jobs is to use the number of cores on your machine. My dev environment VM has 4 cores but I wanted to see what the different values had on run time.

$ pg_dump -Fc db > dumpfc.gz
real 6m28.497s
user 5m2.275s
sys 1m16.637s

$ pg_restore -d db -j 2 dumpfc
real 25m39.796s
user 1m30.366s
sys 1m7.032s

Total time 32 minutes, 7 seconds, (25.6 minutes restoring dev 3% faster than plain pg_restore).

Alright, that’s a tiny improvement, can we push this more?

$ pg_dump -Fc db > dumpfc.gz
real 6m28.497s
user 5m2.275s
sys 1m16.637s

$ pg_restore -d db -j 4 dumpfc.gz
real 22m6.124s
user 0m58.852s
sys 0m34.682s

Total time 28 minutes, 34 seconds (22.1 minutes restoring dev 14% faster than two jobs).

Excellent, specifying four jobs over two speeds up the overall restore time by 14%. At this point, we’ve sped it up from 32.5 minutes on our dev environment to 22.1 minutes, a 32% increase!

My thinking now was how far could I push this number?

$ pg_dump -Fc db > dumpfc.gz
real 6m28.497s
user 5m2.275s
sys 1m16.637s

$ pg_restore -d db -j 8 dumpfc.gz
real 16m49.539s
user 1m1.344s
sys 0m39.522s

Total time 23 minutes, 17 seconds (16.8 minutes restoring dev, 24% faster than four jobs).

So specifying double the number of jobs as cores have reduced the time from 22.1 minutes to 16.8 minutes. At this point, I’ve sped up our entire restore time by 49% which is fantastic.

Can I push this even further?

$ pg_dump -Fc db > dumpfc.gz
real 6m28.497s
user 5m2.275s
sys 1m16.637s

$ pg_restore -d db -j 12 dumpfc.gz
real 16m7.071s
user 0m55.323s
sys 0m36.502s

Total time 22 minutes, 35 seconds (16.1 minutes restoring dev, 4% faster than eight jobs).

Specifying 12 concurrent jobs does speed the process up by a tiny bit but overall affects the VM’s CPU usage to the point where it’s not overly usable while the restore is happening. At this point, I settled on eight jobs or double the number of cores as being the optimal number to use.

Final Thoughts

In the end, our active got cut almost in half, from 30 minutes down to 16 minutes. This saves us over 72 hours of restore time a year (6 devs, 52 restores a year, 14 minutes). Overall I’m very happy with these changes. In the future, I’ll be looking at just restoring data and not the entire DB and see how much faster that is.

In Speeding up Postgres Restores Part 2 we improve the time even further, bringing it down to 5 minutes.

The post Speeding up Postgres Restores appeared first on Procrastinating Developer.

Installation

Installing Clip is simple with pip:
$ pip install clip

Getting the Code

You can either clone the public repository:
git clone git@github.com:silent1mezzo/clip.git

Download the tarball:
$ curl -OL https://github.com/silent1mezzo/clip/tarball/master

Or, download the zipball:
$ curl -OL https://github.com/silent1mezzo/clip/zipball/master

Once you have a copy of the source, you can embed it in your Python package, or install it into your site-packages easily:
$ python setup.py install

Quick Start

You can get started with clip quickly by typing clip in your terminal to pull up the help text.

Here are a few commands you can try out

Creating a List:

$ clip <list_name> # Creates one if it doesn't exist
$ clip websites

Viewing a List:

$ clip <list_name> # If a list exists, view it
$ clip websites

Adding a snippet:

$ clip <list_name>
$ clip websites django3.0 https://docs.djangoproject.com/en/3.0/releases/3.0/

Getting a snippet:

$ clip <list_name>
$ clip websites django3.0 https://docs.djangoproject.com/en/3.0/releases/3.0/
`https://docs.djangoproject.com/en/3.0/releases/3.0/` has been copied to your clipboard

You can also omit the list_name and it’ll try to find the key

$ clip django3.0 https://docs.djangoproject.com/en/3.0/releases/3.0/
`https://docs.djangoproject.com/en/3.0/releases/3.0/` has been copied to your clipboard

Deleting a List/Key

$ clip delete <list_name>
$ clip delete websites
 
$ clip delete <list_name> <key>
$ clip delete websites django3.0

Opening a snippet in your browser:

$ clip open <list_name>
$ clip open websites django3.0
 
$ clip open 
$ clip open django3.0

The post Clip appeared first on Procrastinating Developer.

Getting Started

For the sake of simplicity let’s make our own User object that is exactly the same as Django’s current but fixes the email max_length field to comply with RFC 5321 of 254 characters and adds a required field for the user’s twitter handle.

# myapp.models.py 
from django.contrib.auth.models import AbstractBaseUser
 
class MyUser(AbstractBaseUser):
    username = models.CharField(max_length=40, unique=True, db_index=True)
    email = models.EmailField(max_length=254, unique=True)
    twitter_handle = models.CharField(max_length=255)
    ...
 
    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = ['twitter_handle']

You’ll notice a few new things about this:

Inheriting AbstractBaseUser:
This adds a few helper fields such as password and last_login as well as methods for setting the user’s password, getting the username, checking if the user is active. You can check out a full list of what it includes here.

USERNAME_FIELD:
This is a string describing the name of the field on the User model that is used as the unique identifier. This will usually be a username of some kind, but it can also be an email address or any other unique identifier.

REQUIRED_FIELDS:
A list of the field names that must be provided when creating a user. REQUIRED_FIELDS must contain all required fields on your User model, but should not contain the USERNAME_FIELD.

Now that you’ve created your User model you have to tell Django that you want to use it instead of their default User model. To do this you add the following to your settings file:

# settings.py
AUTH_USER_MODEL = 'myapp.MyUser'

With this setting Django now knows which User model to use.

Using Foreign Keys

Once you’ve set up your model it’s now time reference it in other models.

from django.conf import settings
from django.db import models
 
class Post(models.Model):
    author = models.ForeignKey(settings.AUTH_USER_MODEL)

This tells Django to create a ForeignKey to the User model that you specify in your settings.

Custom Manager

Now that you’ve created your own User model you also need to create your own Manager to handle the creation of Users and Super Users. If your User model defines the same fields as Django’s default User you can just install Django’s UserManager.

from django.contrib.auth.models import UserManager, AbstractBaseUser
 
class MyUser(AbstractBaseUser):
    ...
    objects = UserManager

If your User model includes different fields you’ll need to define your own custom manager that extends BaseUserManager.

from django.contrib.auth.models import BaseUserManager, AbstractBaseUser
 
class MyUserManager(BaseUserManager):
    def create_user(self, email, twitter_handle, password=None):
        if not email:
            raise ValueError('Users must have an email address')
 
        user = self.model(
            email=MyUserManager.normalize_email(email),
            twitter_handle=twitter_handle,
        )
 
        user.set_password(password)
        user.save(using=self._db)
        return user
 
    def create_superuser(self, email, twitter_handle, password):
        user = self.create_user(email,
            password=password,
            twitter_handle=twitter_handle
        )
        user.is_admin = True
        user.save(using=self._db)
        return user
 
 
class MyUser(AbstractBaseUser):
    ...
    objects = MyUserManager

Other Methods

There are a few other methods you need to include:

get_full_name:
A longer formal identifier for the user. A common interpretation would be the full name of the user, but it can be any string that identifies the user.

get_short_name:
A short, informal identifier for the user. A common interpretation would be the first name of the user, but it can be any string that identifies the user in an informal way.

is_active:
A boolean attribute that indicates whether the user is considered “active”. This attribute is provided as an attribute on AbstractBaseUser defaulting to True.

Final Example

# models.py
from django.conf import settings
from django.db import models
from django.contrib.auth.models import BaseUserManager, AbstractBaseUser
 
class MyUserManager(BaseUserManager):
    def create_user(self, email, twitter_handle, password=None):
        if not email:
            raise ValueError('Users must have an email address')
 
        user = self.model(
            email=MyUserManager.normalize_email(email),
            twitter_handle=twitter_handle,
        )
 
        user.set_password(password)
        user.save(using=self._db)
        return user
 
    def create_superuser(self, email, twitter_handle, password):
        user = self.create_user(email,
            password=password,
            twitter_handle=twitter_handle,
        )
        user.is_admin = True
        user.save(using=self._db)
        return user
 
 
class MyUser(AbstractBaseUser):
    email = models.EmailField(max_length=254, unique=True, db_index=True)
    twitter_handle = models.CharField(max_length=255)
 
    is_active = models.BooleanField(default=True)
    is_admin = models.BooleanField(default=False)
 
    objects = MyUserManager()
 
    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = ['twitter_handle']
 
    def get_full_name(self):
        # For this case we return email. Could also be User.first_name User.last_name if you have these fields
        return self.email
 
    def get_short_name(self):
        # For this case we return email. Could also be User.first_name if you have this field
        return self.email
 
    def __unicode__(self):
        return self.email
 
    def has_perm(self, perm, obj=None):
        # Handle whether the user has a specific permission?"
        return True
 
    def has_module_perms(self, app_label):
        # Handle whether the user has permissions to view the app `app_label`?"
        return True
 
    @property
    def is_staff(self):
        # Handle whether the user is a member of staff?"
        return self.is_admin
 
 
class Post(models.Model):
    author = models.ForeignKey(settings.AUTH_USER_MODEL)
 
 
#views.py
from django.contrib.auth import get_user_model
...
User = get_user_model()

The post Using Configurable User Models in Django 1.5 appeared first on Procrastinating Developer.