Procrastinating Developer » Programming http://procrastinatingdev.com Random Musings about Coding, Food and Beer Fri, 02 May 2014 12:46:01 +0000 en-US hourly 1 https://wordpress.org/?v=4.3.17 Getting Ready for a Hackathon http://procrastinatingdev.com/programming/getting-ready-for-a-hackathon/ http://procrastinatingdev.com/programming/getting-ready-for-a-hackathon/#comments Wed, 27 Mar 2013 16:00:38 +0000 http://procrastinatingdev.com/?p=1123 Continue reading Getting Ready for a Hackathon]]> Three or four times a year I take part in hackathons for various reasons such as networking, learning new technical skills and getting new ideas on things to work on. Every time I go to a hackathon I have a list of things that I do to get ready and I thought I’d share them with you today.

Personal Checklist:

  • Water
  • Clothes (single day/multiple days)
  • Sleeping Bag/Pillow (if overnight)
  • Advil/Tylenol
  • Deodorant
  • Toothbrush and Toothpaste
  • Business Cards

Hardware Checklist:

  • Laptop and Charger
  • Phone and Charger
  • Projector Adapter
  • Headphones
  • Pens/Pencils and Paper
  • Keyboard and Mouse

Software Checklist:

  • Technical Stack (VM’s, local machine, favourite programming language)
  • Github Repository
  • Domain Name (or at least a registrar you’re familiar with)
  • Server (Heroku/Webfaction)
  • Any SDK’s or Frameworks installed
  • Agreed upon standard tools (IRC, Editor, Trello, etc…)

Idea Checklist:

  • Basic Idea
  • Knowledge of any needed APIs
  • Checklist of Features

I’ve shown up with just a laptop and a charger before and still placed relatively high. Planning before hand just gives you that little extra advantage over the other people there. What tools/things do you bring to a hackathon?

]]>
http://procrastinatingdev.com/programming/getting-ready-for-a-hackathon/feed/ 1
Crafting your Conference Talk Proposal http://procrastinatingdev.com/programming/crafting-your-conference-talk-proposal/ http://procrastinatingdev.com/programming/crafting-your-conference-talk-proposal/#comments Wed, 23 Jan 2013 16:11:48 +0000 http://procrastinatingdev.com/?p=1118 Continue reading Crafting your Conference Talk Proposal]]> With the voting period over for DjangoCon Europe talk proposals I wanted to take a couple of minutes and go over what I think makes a good conference talk proposal and a couple of things that you shouldn’t do.

Keep it Informative

A lot of people try to be funny in their proposals but miss out on the key points. Your proposal should include a brief description of what you’ll be talking about, the audience that it’s intended for and anything else that you think would be useful for the audience.

Keep it Short

Conferences get a lot of talk proposals. PyCon US got 450 submissions for 114 slots. DjangoCon Europe got 74 submissions for 20 slots. With such competition the reviewers have to go over a lot of content. It’s important to keep your proposal short and sweet so you don’t bore the reviewers and risk them skipping over your talk.

I haven’t done any statistical analysis but a brief look over the proposals for DjangoCon EU makes it look like the sweet spot for proposal length is 4-7 sentences. Shorter and the reviewers can’t get enough relevant information, longer they get bored and go on to the next one.

Keep it Relevant

Development communities seem to have an almost cyclical nature when it comes to “hot topics”. One year it’ll be deploying, another year your software stack and the next year best practices for caching. Some topics, such as performance, databases and security are always fresh. Make sure your talk is relevant, both to yourself, and the community you’re presenting to.

Stop Rehashing Talks

We get it, Django sucks, this other community is better than ours, your side project is the next best thing. These talks have been done a million times and while they sometimes add a new twist or new content for the most part they’re old news. If you’re going to do a talk that’s been done before make sure it’s done in a new way with new material.

This year I submitted a proposal on “Caching Django” for DjangoCon Europe which ended up ranking 7th out of 74 proposals. I’m looking forward to hopefully going to Warsaw Poland and giving my talk.

]]>
http://procrastinatingdev.com/programming/crafting-your-conference-talk-proposal/feed/ 5
Why I Became a Programmer http://procrastinatingdev.com/programming/why-i-became-a-programmer/ http://procrastinatingdev.com/programming/why-i-became-a-programmer/#comments Mon, 07 Jan 2013 18:40:58 +0000 http://procrastinatingdev.com/?p=1106 Continue reading Why I Became a Programmer]]> When I was in grade 9 I wanted to be a lawyer. I didn’t really know what that entailed and I had no idea how many more years of schooling I’d have to complete to realize my dream. In grade 10 I took my first law class and absolutely hated my teacher. From that moment on I knew I would never become a lawyer. I dropped out of law class and took computer science and fell in love. Over the years I’ve learnt more and more and I thought I’d take a few minutes to talk about “Why I Became a Programmer”.

Problem Solving

For as long as I can remember I’ve loved solving puzzles. When I would get a new puzzle I’d try to break it down to smaller logical chunks to try to see how the puzzle worked. That’s exactly what I get to do at work every day. I first start with a large overarching problem that’s given to me by a customer and then I have to break it down into smaller logical chunks. Sometimes these problems aren’t exciting or challenging. I’ve written enough CRUD websites that they no longer give challenges but sometimes these problems are unique and fun.

Building Stuff

While I love solving problems and it’s fun to have a challenging career I really enjoy building things. As a kid I was always taking stuff apart, finding out how it worked and then building it back. Writing software provides me with quick gratification because I can see the immediate results of my work. I can quickly put together a new app and once deployed see it being used instantly on the Internet.

Being a programmer also lets me build a large range of things. I can build a website one day, an iPhone app another and configure a sever on the weekend. This diversity allows programming to feel new and exciting even if you’ve done it for a few decades.

Continuous Learning

As a programmer, if you’re not learning the latest technology you get left behind very quickly. There’s always a new language coming out or a new framework to read about. While this can become overwhelming at first I often find trying a new language out to be a fun experiment on a day off. I’ve also found that this constant learning makes programming stay fresh and fun.

These are the main reasons why I became a programmer but I’d love to hear why others did. Let me know in the comments below.

]]>
http://procrastinatingdev.com/programming/why-i-became-a-programmer/feed/ 1
Announcing Clip – Your Easy CLI Clipboard Manager http://procrastinatingdev.com/programming/announcing-clip-your-easy-cli-clipboard-manager/ http://procrastinatingdev.com/programming/announcing-clip-your-easy-cli-clipboard-manager/#comments Fri, 21 Dec 2012 16:51:30 +0000 http://procrastinatingdev.com/?p=1087 Continue reading Announcing Clip – Your Easy CLI Clipboard Manager]]> Yesterday I finished up the first version of Clip, the easy command-line interface clipboard manager. Currently it’s OS X only but I plan to make it platform agnostic over the coming weeks. Here’s how you can get started using Clip.

Installation

Installing Clip is simple with pip:
$ pip install clip

You can also use a mirror by typing:
$ pip install -i http://simple.crate.io/ clip

Getting the Code

You can either clone the public repository:
git clone git@github.com:silent1mezzo/clip.git

Download the tarball:
$ curl -OL https://github.com/silent1mezzo/clip/tarball/master

Or, download the zipball:
$ curl -OL https://github.com/silent1mezzo/clip/zipball/master

Once you have a copy of the source, you can embed it in your Python package, or install it into your site-packages easily:
$ python setup.py install

Quick Start

You can get started with clip quickly by typing clip in your terminal to pull up the help text.

Here are a few commands you can try out

Creating a List:

$ clip <list_name>
$ clip websites

Viewing a List:

$ clip <list_name>
$ clip websites
...

Adding a snippet:

$ clip <list_name>  
$ clip websites django1.5 https://docs.djangoproject.com/en/dev/releases/1.5/

Getting a snippet:

$ clip <list_name> 
$ clip websites django1.5
'https://docs.djangoproject.com/en/dev/releases/1.5/' has been copied to your clipboard

You can also omit the list_name and it’ll try to find the key

$ clip django1.5
'https://docs.djangoproject.com/en/dev/releases/1.5/' has been copied to your clipboard

Deleting a List/Key

$ clip delete <list_name>
$ clip delete websites
 
$ clip delete <list_name> 
$ clip delete websites django1.5

Opening a snippet in your browser:

$ clip open <list_name> 
$ clip open websites django1.5
 
$ clip open 
$ clip open django1.5

 

Hope you enjoy using Clip as much as I do!

]]>
http://procrastinatingdev.com/programming/announcing-clip-your-easy-cli-clipboard-manager/feed/ 1
Creating your first Blog http://procrastinatingdev.com/programming/creating-your-first-blog/ http://procrastinatingdev.com/programming/creating-your-first-blog/#comments Mon, 13 Aug 2012 21:00:56 +0000 http://procrastinatingdev.com/?p=905 Continue reading Creating your first Blog]]> Starting a blog is an exciting time. Ideas for potential themes, posts and frameworks running through your head. If you’re a technical user you’ll already have bought your domain name, forwarded it to your hosting provider and started creating your own theme. If you’re like most people though you probably don’t have the faintest idea on how to get started. Hopefully after reading this post you’ll have a better idea on how to create your first blog.

Buying a Domain

The first step to starting a blog is finding a relevant domain name that hasn’t been taken. Using services like domai.nr and instantdomainsearch.com can help with your search.

Once you’ve found a domain name that isn’t taken you’ll have to register it with a domain registrar. Websites like netfirms.com and name.com will allow you to buy a domain name for under $10.

Finding a Hosting Provider

Once you’ve bought your domain you have to find a place to host it. While there are many ways to host your website one of the easiest and least expensive options I’ve found is webfaction. Not only do they give you the ability to host your blog but they make it extremely easy to do so. Some hosts you’ll have to manually setup WordPress (or another blogging framework) and use a command line to configure things. With webfaction you simply use their website to create a WordPress site.

Finding a Theme

So you’ve bought your domain, found a place to host it at and are now ready to personalize your website. If you’re familiar with CSS/HTML and PHP you could make your own theme and spend hours tweaking the colours or, if you’re like me, you could use a pre-made theme. There are a number of market places but my favourite is themeforest.net. In my opinion they have the best variety and quality.

If you don’t feel like paying for a theme you can also use one of the two defaults that come with WordPress or look for a free one from WordPress or you can search for one through WordPress Admin.

Once you’ve found the theme you want to use you can add it in just a few steps:

  1. Log into WordPress Admin (www.your-domain.com/wp-admin)
  2. Click on Appearance -> Themes
  3. Click on Install Themes
    4a. If you haven’t found a theme yet you can search from this screen.
    4b. If you’ve already bought/downloaded a theme click ‘Upload’
  4. Click choose file
  5. Find the .zip file for the theme and select it.
  6. Click Install Now

Once you’ve followed those steps you should have your theme installed and ready for your first post.

Creating your first Page

Pages are static content that are often written when you first set up your blog and then left alone for a long time. An example of the page is my About page where I’ve listed relevant information about myself. Other examples are Contact pages or an Event page.

To write your first page, follow these steps:

  1. Log into WordPress Admin
  2. Click Pages -> Add New
  3. Enter in the page Title
  4. Enter the page content
  5. Click Publish
Pages often show up at the top of your theme or, in my case, along the side.

Writing your first Post

Posts are where you write your daily thoughts, tips and content for your visitors. By default categories show up on the front page of your blog with the latest posts showing up at the top. If you want, you can assign categories to posts so they show up in different spots or to make it easier for your visitors to find different type of posts.

To write your first post, follow these steps:

  1. Log into WordPress Admin
  2. Click Posts -> Add New
  3. Enter the post Title
  4. Enter the post content
  5. Select a category (or leave it blank for the default)
  6. Fill in any other information that you need (plugins can add more options below the post content).
  7. Click Publish
Those are the steps that you need to create a post. With this information you can pretty much do everything you need with WordPress and get your blog started.
]]>
http://procrastinatingdev.com/programming/creating-your-first-blog/feed/ 1
A Month in Review http://procrastinatingdev.com/programming/a-month-in-review/ http://procrastinatingdev.com/programming/a-month-in-review/#comments Tue, 20 Sep 2011 10:00:30 +0000 http://procrastinatingdev.com/?p=652 Continue reading A Month in Review]]> As of today ProcrastingDev.com officially turns 1 month old. I know a lot of websites do monthly/yearly reviews on their traffic and I wanted to do the same. I’m planning on doing this on a month by month basis to try to get an understanding on how well I’m doing promoting the blog.

Here are the first months statistics:

Number of Articles: 36
Number of Comments: 8
Visitors: 2,801 visitors
Page Views: 5,442 pageviews (1.94 pages/visit)
Average Time: 2:04 min. avg time on site
Bounce Rate: 63.87%
RSS Subscribers: 19

 

Top Sources:

Direct – 1,067 visits (38.17%)
Feedburner – 686 visits (24.57%)
Google (Referrer) – 306 visits (11.01%)
Google (Search) – 183 visits (6.33%)
Twitter – 148 visits (5.22%)

 Top Content:

Procrastinatingdev.com
Making Interactive Maps for the Web
Advanced Django Form Usage
Building APIs in Django with TastyPie
Real Word Deployment using Chef

 

DjangoCon 2011

As you can see from the Top Content for this month all of the posts are my live-blogging from DjangoCon. A lot of my traffic came from live-blogging the conference because the feeds showed up on Django’s main website, on a lot of people’s twitter feeds and all over the place. While the conference is now over I’m still getting a lot of residual search engine traffic from these articles.

The big spikes in the traffic are during that conference as people searched for #djangocon2011. The blog also got mentioned by a lot of high-profile Django developers so that helped as well.

Promotion

So far, other than adding my feed to a couple of feed aggregators and tweeting out my articles I haven’t really done any promotion. When I comment on different tech blogs I include the URL when available but the traffic from this is very minimal.

In the future I’d like to start guest-blogging for larger sites to try to gather a larger following. This is something I’ll try this month.

Final Thoughts

DjangoCon certainly gave me a boost in traffic and while it has decreased now that the conference is over the overall traffic is still greater (3-4x) than before the conference.

I’m going to try to continue posting every Monday, Wednesday and Friday as this seems to be working.

Stay tuned for more goodness…

]]>
http://procrastinatingdev.com/programming/a-month-in-review/feed/ 2
Write Code as if People Will Read It http://procrastinatingdev.com/programming/write-code-as-if-people-will-read-it/ http://procrastinatingdev.com/programming/write-code-as-if-people-will-read-it/#comments Mon, 12 Sep 2011 06:00:29 +0000 http://procrastinatingdev.com/programming/write-code-as-if-people-will-read-it/ Continue reading Write Code as if People Will Read It]]> “Oh I’ll just write this little script to do. Nobody will ever see it”. I have heard many programmers say that exact sentence. And inevitably someone at some point will see the hacked together script that you never meant for public consumption.

I want to propose a challenge to everyone. Write code as if everyone in the world will see it. Here are a few things you can do to complete this challenge.

  1. Use comments sparingly
    Comments are excellent ways to give people a quick understanding of what a specific function does or to give insight on a given file. Be careful with having too many comments. Over saturating your code with comments can be worse than not having any comments. It can make your code difficult to read and obfuscate rather than help.
  2. Use meaningful variable names
    This one is pretty self-explanatory. When you use single letter variable names you’ll quickly forget the meaning behind them and then have scour the source code to find out what they do.
  3. Don’t use overly complex statements
    Simple is almost always better, especially when it comes to readability. One liners are great for compressing your code but in most cases they’re harder to read (especially for beginner programmers).

    # In Python you can do
    x if condition else y
    # Or you can also do
    (lambda:y, lambda:x)[condition]()
     
    # Both of the above can be difficult for beginner programmers.
    # Using a simple if/else statement is easier
    if condition:
        x
    else:
        y
  4. Use whitespace properly
    Whether you use Python where whitespace is mandatory or languages like C or Java where it isn’t, using whitespace properly greatly improves readability. Programs like this one lose a lot of their readability by not using whitespace.
  5. Have proper unit tests
    I can’t stress this one enough. If you have valid unit tests it makes it a lot easier for people (and yourself) to use. Writing unit tests will make sure that you code runs properly even when you add new functionality. It’ll also make changing existing functionality easier.
  6. Be able to explain it in 5 minutes or less
    This tip may seem a little weird but you should be able to explain your code in 5 minutes or less. If it takes longer, your code may be too complex. Splitting it up into extra functions, methods or files could help. If you can’t explain what the code does then it’s unlikely that somebody else will be able to figure it out by reading it.Being able to explain your code also helps you organize your thoughts and decide whether or not the way you’re doing something is correct.

To sum everything up. Always write your code (even your little scripts) as if somebody else will be reading it. This will help you be a better programmer and will make it easier when you need to come back to the code in the future.

]]>
http://procrastinatingdev.com/programming/write-code-as-if-people-will-read-it/feed/ 0
DjangoCon 2011 – Deployment, Daemons and Datacenters http://procrastinatingdev.com/programming/djangocon-2011-deployment-daemons-and-datacenters/ http://procrastinatingdev.com/programming/djangocon-2011-deployment-daemons-and-datacenters/#comments Thu, 08 Sep 2011 21:16:55 +0000 http://procrastinatingdev.com/?p=541 Continue reading DjangoCon 2011 – Deployment, Daemons and Datacenters]]> Following a great talk on Security in Django we now have “Deployment, Daemons and Datacenters” by Andrew Godwin. This talk will go into the deployment strategies at ep.io.

A tour through the systems that power ep.io, the Python hosting platform, from the array of daemons powering the system, to how redundancy is set up, and also covering general best practices for hosting Django sites yourself.

Updates Below:

17.58

End of the talk. I’ll try to post the slides once they’re available.

17.58

“Have you dealt with the fact that different people have different laws with regards to Backups?”

In the EU you cannot export data outside of the country. You need to be aware of the different laws. Often good idea to keep backups in the same country, but different cities.

17.56

“Why did you move from AWS to your own hardware?”

A combination of performance. When you looked at the cost per dollar it was cheaper to have their own hosted service. Disk IO is a lot faster on their own hardware than virtualized hosts.

17.55

“How do you handle file uploads with multiple application servers?”

Uses http://www.gluster.org/ on a shared server.

17.54

Andrews done talking. Question time.

17.51

Standard tools aren’t always the best.

  • ep.io load balancer was initially HAProxy (doesn’t like having 3000 backends being reloaded every 10 seconds)
  • Custom eventlet-based load-balancer was simpler and slightly faster

17.49

Automation

  • Use Puppet or Chef along with Fabric
  • If you do something more than three times, automate it
  • Everything you manually SSH in, a kitten gets extremely worried

17.49

Loose Coupling

  • Simple, loosely-connected components
  • Easier to test and easier to debug
  • Enforces some rough interface definitions

17.48

Plan for multiple machines
  • That means no SQLite in production (doesn’t work for multiple machines)
  • Make good use of database transactions
  • How are you going to store uploaded files?

17.47

Sensible Architecture

Ship long-running tasks off:

  • Use celery, or your own worker solution
  • Even more critical if you have syncrhonous worker threads in your web apps
  • Email sending can be very slow

17.46

An easy start…

  • Dump your database nightly to a SQL file
  • Use rdiff-backup (or similar)_ to sync your DB dump, codebase and uploads to a backup directory
  • Also sync offsite – get a VPS with a different provider than your main one
  • Make your backup server pull the backups, don’t push them to it.

17.44

Replication is Hard

  • PostgreSQL and Redis replication both require your code to be modified a bit
  • Django offers some help with database routers
  • It’s also not always necessary and can cause bugs for your users (small sites may not be the answer)

17.42

Check your backups restore

  • Just seeing if they’re there isn’t good enough
  • Try restoring your entire site onto a fresh box

17.42

Backups before any major change in the database or code.

“It’s tedious but the one time you need it it’ll help you”

17.41

Never back up to the same provider. They can cancel your account…

17.40

Backups and Redundancy

Archives != High Availability

  • Your PostgreSQL slave is not a backup
  • You should backup using multiple formats to diverse locations

17.39

Development and Staging

  • No need to run gunicorn/nginx locally (runserver still works)
  • PostgreSQL 9 still slightly annoying to install
  • Redis is very easy to set up
  • Staging should be EXACTLY the same as live

17.36

How to handle higher loads:

  • Varnish for site caching
  • HAProxy or Nginx for load-balancing
  • Give PostgreSQL more resources

17.36

ep.io Stack

Three years ago

  • Apache and mod_wsgi
  • PostgreSQL 8.x
  • Memcached
Today
  • Nginx (static files/gzipping)
  • Gunicorn (dynamic pages, unix socket best)
  • PostgreSQL 9
  • Redis
  • virtualenv

17.34

Security

  • ep.io treas their internal network as public (any traffic has to be signed/encrypted)
  • Firewalling of unnecessary ports
  • Separate machines for higher-risk processes

17.33

The Joy of Networks

  • Any network has a significant slowdown compared to local access
  • Locking and concurrent access also an issue
  • Internal latency on EC2 can peak higher than 10s
  • Routing blips can cause very short outages

17.31

More ep.io statistics

15 requests, some git some pypi

  • Traditional: 300 seconds
  • Parellised, no cache: 30 seconds
  • Parellised, cache: 2 seconds

17.30

They run a parallel version of pip (with caching). Not 100% compatible with complex dependencies

17.29

Some ep.io information

  • Everytime an app is uploaded to ep.io it gets a fresh app image to deploy into
  • Each app image has its own virtualenv
  • The typical ep.io has around 3 or 4 dependencies
  • Some have more than 40

17.28

Using ZeroMQ and Eventlet works well together.

17.26

Eventlet is…

  • Coroutine-based asynchronous concurrency
  • Basically, lightweight threads with explicit context switching
  • Reads quite like procedural code

17.26

Redundancy’s not easy.

Serveral things can only run once (cronjobs)

17.25

sock = ctx.socket(zmq.REQ)
for endpoint in self.config.query_addresses():
    sock.connect(endpoint)

payload = json.dumps({'type':type, 'extra':extra})

with Timeout(30):
    sock.send(self.sign_message(payload))
    return self.decode_message(sock.recv())

17.23

ZeroMQ & Redundancy

  • Not a message queue
  • Advanced sockets, with multiple endpoints
  • Has both deliver-to-single-consumer and deliver-to-all-consumers
  • Uses TCP for transport

17.22

What’s ep.io?

  • Hosts Python sites/daemons
  • Technically language-independent
  • Supports multiple kinds of databases

17.21

Andrew is taking the stage.

Andrew is…

  • Core Developer
  • South author
  • Cofounder of ep.io

17.17

Everyone just getting settled. Stay tuned.

]]>
http://procrastinatingdev.com/programming/djangocon-2011-deployment-daemons-and-datacenters/feed/ 0
DjangoCon 2011 – Secrets of PostgreSQL Performance http://procrastinatingdev.com/programming/djangocon-2011-secrets-of-postgresql-performance/ http://procrastinatingdev.com/programming/djangocon-2011-secrets-of-postgresql-performance/#comments Tue, 06 Sep 2011 23:18:14 +0000 http://procrastinatingdev.com/?p=385 Continue reading DjangoCon 2011 – Secrets of PostgreSQL Performance]]> After a slightly dry but informative talk from Russell Keith-Magee on the current state of the Django Software Foundation (which I forgot to live-blog) I’m back with the next talk of the night by Frank Wiles

PostgreSQL is effectively the default RDBMS for Django. Learn the dark arts of optimizing this powerful database to be blazingly fast on your own hardware or in the cloud.

You can view the slides at http://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf

Updates below:

19.52

Frank has finished talking now. Extremely well thought out talk. Good combination of technical tips and general insight.

19.48

Break large tables into smaller tables by date or another partitioning scheme that fits your data.

19.47

Using Triggers can be quicker than Signals. Consider using if you do a lot of repetitive stuff.

19.47

Statistics

default_statistics_target = 10

ALTER <table> ALTER <column> SET STATISTICS <integer>;

19.46

Performance Tools:

  • Django Debug Toolbar – Easily see # of queries and diagnose problems
  • Slow Query Log (log_min_duration = 1000)
  • pgfouine – Will show you the most frequent by time queries
  • EXPLAIN

19.44

Partial Indexes:

CREATE INDEX open_tickets_idx ON tickets (OPEN) WHERE OPEN='t';

19.43

Multicolumn Indexes can help if you’re constantly doing searches on multiple columns.

19.41

Instead of using count() you can have a simple key-value table that has a count of your counts. Update via triggers in PG or save signals

19.40

Common Django Specific problems:

  • Use select_related()
  • Queries in loops (STAY AWAY!)
  • Proper indexing (Index the proper things, don’t index columns you select infrequently)
  • count() is slow. PG has statistics on it’s tables which you can use instead.

19.37

Mount your filesystem with noatime

19.36

“Your choice of filesystem matters. Don’t use a journaled filesystem for your WAL. Use XFS for maximum performance”

19.34

When working “In the Cloud”

  • Remember: You’re sharing disks and you don’t even know how!
  • You can get the best of both worlds with Rackspace and their Cloud Connect product
  • Using EBS volumes and software RAID is best (but somewhat scary) option on AWS

19.33

“If you can afford SSDs, you can ignore the rest of this talk”

19.32

Use Tablespaces:

  • Allows you to specify where you want specific tables, indexes on different disks/volumes
  • Consider putting archive or legacy data onto slower drives/volumes
  • Separate your data and indexes onto different volumes (if possible)

19.31

Use pgbouncer to pool connections

19.30

Hardware Considerations:

  • Have as much RAM as possible (best bang for your buck)
  • Disks, have lots of disks (Faster disks make a huge difference. Raid 1+0 is good, Raid 5 bad. Separate out the WAL onto it’s own disk)
  • After previous two worry about CPU speed. You’re unlike to be CPU bound

19.28

Other parameters

wal_buffers – set to 16MB and forget it

checkpoint_segments – Increase to at least 10

maintenance_work_mem – 50MB for every GB of Ram (stats tools)

synchronous_commit – Turn off with data loss risks (Potential for losing some data. Default is it’ll pool connections for every 600ms)

19.25

The “Big 3” Tuning Parameters

shared_buffers – Set to 25% of available RAM and move up/down 5% to find the sweet spot

effective_cache_size – Planning hint that tells PG how much RAM it can expect for OS disk cache

work_mem – Per processes amount of ORDER BY space

19.24

“Do Smart Things”

  • Cache and Cache some more
  • Watch your query counts (1.3 has assert num queries, use it!)

19.23

“Don’t do dumb things”.

Don’t…

  • Assume PostgreSQL is like MySQL
  • Database server doing double duty
  • Disk Contention
  • Retrieve more data than you need

19.21

“Measure Everything Before and After you try it”

19.21

And we’re off. Frank Wiles takes the stage.

]]>
http://procrastinatingdev.com/programming/djangocon-2011-secrets-of-postgresql-performance/feed/ 0
DjangoCon 2011 – Testing: The Developer Strikes Back http://procrastinatingdev.com/programming/djangocon-2011-testing-the-developer-strikes-back/ http://procrastinatingdev.com/programming/djangocon-2011-testing-the-developer-strikes-back/#comments Tue, 06 Sep 2011 21:18:07 +0000 http://procrastinatingdev.com/?p=343 Continue reading DjangoCon 2011 – Testing: The Developer Strikes Back]]> Next up is “Testing: The Developer Strikes Back” by Sandy

“Code not tested is broken by design” – Jacob Kaplan-Moss Every programming project has unique testing needs. It’s not always easy to understand what those needs are, or how to write tests that satisfy them. One of our goals as developers is to write tests that minimize failures in production that can cost our companies time, money, resources, and in many cases, hours of sleep!

You can view Sandy’s talks at https://docs.google.com/present/view?id=0AVthC0Z3iw8DZGRrdnFzeGdfN2c5bWJ6d2Y1&hl=en_US

Updates below:

18.01

Tests don’t replace good documentation

17.59

When writing tests, pretend that you’re a user and test against what they would be doing.

17.56

How do you train your developers between “My test suite passes” and “My test suite actually tests my code”?

Make sure your developers look at the code. Break the code and see if the code still passes. Trying to break the test suite.

17.54

@sandymahalo talk over now. Fantastic speaker, answering questions now.

17.53

How to sell testing at your Django shop

  • Saves money
  • Saves development time ($$$)
  • Happier developers
  • Saves on QA time ($$$)
  • Gives you more confidence in your code

17.51

Staging environment should be identical to production (or as close as possible)

17.51

Code should not be written in such a way that it won’t work (at least in some fashion) if a certain service or infrastructure fails

17.49

Service unavailable shouldn’t mean that your site is unavailable.

17.49

Developers need to think outside of their dev instances. What happens to the code under condition X when it interacts with service Y

17.48

Fixtures are great for loading data required to bootstrap your application. ObjectCreator classes for models that change frequently.

17.47

Testing a virgin codebase:

  • Test Driven Refactoring
  • Broadly test each app. Drill down testing afterwards
  • Require unit tests for all code going forward
  • Reserve time/points to clear out “test debt”
  • Establish a good foundation to build on
  • Every bug is two bugs: One in the code that broke, and one in the test that failed to detect it
  • Implement Continuous Integration (CI) to monitor coverage

17.44

Sometimes testing can difficult because it can depend on your environment.

17.42

Make a game out of testing with coverage.py

17.41

Testing allows for faster iterations and deployments with more confidence. Less 2AM fires that you have to fix.

17.41

Tests are not “set it and forget it”. Tests always need to evolve as new functionality is added

17.40

Well-tested code is a good medium. It’s more realistic, practical and allows for a more individual “style” in the dev cycle

17.39

Test Driven Development (TDD) doesn’t really exist in real life. Deadlines, non-technical management, poor planning, competing obligations get in the way.

17.38

If the tests aren’t easy to write, your functions are too difficult. Try to refactor if possible.

17.37

“Writing tests can improve coding habits”. Smaller, more module code is testable, 200-line functions aren’t

17.36

How do I test cache?
You need to know and understand how your modified application behaves when it interacts with a cache that was generated by your application in a previous state.

17.33

Don’t take 3rd party APIs for granted. Even Facebook goes down from time to time. Have tests to cover what happens when that API is down (or changes)

17.31

http://python-mock.sourceforge.net/ can be used to mock an actual object.

17.31

Using a ObjectCreator class makes the supporting code for your test more loosely coupled and easier to write

17.29

Aggregating certain test methods that are used throughout the entire project are fine to put in a test.py at project root, just don’t put all tests in there.

17.28

Example project layout

project/
    app/
        tests/

17.26

Have multiple test files, don’t aggregate all tests into a monolithic file named test.py

17.25

Separate code and service/infrastructure testing.

17.25

Separation should exist between the testing of:

  • Business logic
  • Datastores
  • 3rd party APIs

17.24

Within a Django project each app needs its own test module.

17.24

What is “Unit testing”

A method by which individual units of source code that are tested to determine if they are fit for use.

17.23

“Testing: It’s hard to do (right)”. Tests are living code and evolve over time

17.22

Ah, its Sandy Strong (@sandymahalo) from PyLadies

17.20

First slide looks like it’s straight from reddit.com. Awesome

17.18

Sandy taking the stage. No idea what her last name is.

]]>
http://procrastinatingdev.com/programming/djangocon-2011-testing-the-developer-strikes-back/feed/ 0