Posted on No Comments on How to optimize a website in 2018

How to optimize a website in 2018

Let’s start with caching. So many levels of caching.

Before we even get to your website, there’s the DNS. What IP does your domain point to? Usually your OS will cache this DNS lookup for you, failing that your DNS server (e.g. your ISP, 8.8.8.8 or 1.1.1.1) will. This isn’t usually something you need to worry about as a web-developer.

Next, the low-hanging fruit: CSS and JS. This is easy to cache. The content is static (rarely changes). We can easily set some HTTP headers via nginx or apache to cache these for a long time. But about when the content does change? Can we afford to wait a week or month before our customers’ browsers pick up the new copy? Can we ask them to press Ctrl+F5? This isn’t a good solution. The easiest and most robust way to handle this is to change the actual filename of the CSS and JS files. If the filename changes, the browser will be forced to download a new copy. Webpack can do this for you, but hooking up the generated filenames into your server-side templating language is a little bit trickier. Generally what it boils down to is using some kind of “webpack stats” plugin to spit out a JSON file listing all of your assets, then have you language of choice read this file and generate the necessary HTML. Not too bad.

Okay, but what about the HTML? Can we cache that too? That gets even harder. Usually the HTML contains data — information that changes frequently. Even if we tried to cache that, how would we cache-bust? Unlike CSS and JS, we can’t change the URL. What if we removed all of the data from the page, and just served templates? The page could run an AJAX request to pull in the data. Better yet, we can use a service worker so that our page loads, even when offline! We can check for new resources in the background and either load them when the user comes back, or encourage them to refresh at their convenience.

What about the data though? First we have to wait for the initial page to load before we can even send out our AJAX request? That’s not cool. Maybe HTTP/2 push can help here? Is it possible to cache the data? We can use IndexDB to store a local copy, and then sync any updates back to the server with Background Sync. But what about sensitive or shared data? We wouldn’t want to send all our data to the client. Even if we managed to send only the stuff they have access to, what would we do if their access is revoked? Too bad — all they have to do is turn-off wifi and they can keep using the app and accessing all the data thanks to PWA.

That’s about all the client-side caching we can do. What about the server though? Where does this data live? In a MySQL server? How long do those queries take? Can those be cached in memory? What if two users are requesting the exact same piece of information at the exact same time? Should we run the query twice? Can they be batched? What if one user is making many requests? Can we combine some of those HTTP requests? How do we know when the data becomes stale? Can we track mutations?

That’s it for caching. Now we have the perfect server setup, we’ve thought of everything, we have an infallible cache-busting strategy, our payloads are under 14KB, but our server just crashed. Oh-oh! I hope you have automatic fail-over. Serving static content from another server isn’t too hard, but data is kind of tricky. Keeping all our database servers in sync in real-time is hard enough, but database writes are even harder.

Oh, there’s also “critical path CSS”, “14KB packets”, optimizing JS for performance and parse time (size isn’t the only thing that’s important). We can put our JS into webworker threads to avoid blocking the main UI thread. Or we can rewrite all our JS in C and compile it to WebAssembly — but don’t do too much interop or you’ll negate the benefit (passing data back and forth isn’t free). What about when your web-app loses focus? Can we scale back some of the renders or network requests to play nice?

These are just some of the things off the top of my head that we as web-developers have to worry about when building a performant website today in 2018.

Posted on No Comments on How to generate a deployment key for Bitbucket

How to generate a deployment key for Bitbucket

  1. SSH into your server
  2. Run “ssh-keygen -t rsa” to generate a public and private key pair
  3. It will ask you where you to save the files.  I recommend “/root/.ssh/bitbucket”
  4. It will ask you to choose a passphrase. Since we’re just using this for deployments, you can leave the passphrase blank. If a hacker manages to steal these files, they probably have access to the code on your server anyway (which is all that this key will give them access to).
  5. Run “cat /root/.ssh/bitbucket.pub”. It will spit out your newly generated public key. Copy it to your clipboard.
  6. On bitbucket.org, go into your repo settings, click “Deployment keys” on the left and paste in your new key
  7. Run “nano /root/.hgrc”
  8. Add this to the file: 
    [ui]
ssh = ssh -i /root/.ssh/bitbucket

[trusted]
users = www-data
  9. cd into your project directory
  10. “hg clone ssh://[email protected]/NAME/PROJECT .” (The trailing dot clones into the current directory if you’ve already created it — make sure it’s empty or this won’t work)
  11. It should clone succesfully now. You can “hg pull && hg up –clean” whenever you like without any passwords. This might not be the most robust way to deploy, but it’s quick and easy.

Substitute “root” for your username if you SSH in as someone else.

 

 

 

 

Posted on No Comments on Meteor, Blade + Windows 8

Meteor, Blade + Windows 8

Introductions

Meteor is a hot new Node-based web development framework.

Blade is a templating language based on Jade, which is a bit similar to Haml.

Windows 8 is an operating system that no one but me likes.

Installation

I’m assuming you’re already up and running with Node.

Meteor isn’t officially supported on Windows yet, so you have to install it a bit differently than suggested on their docs. Fortunately, someone has created an .msi installer for us. Run it.

If you haven’t done so,

npm install -g blade

. If “npm” isn’t a registered command, check your Start Screen for “Node.js Command Prompt” and use that instead of that standard

cmd.exe

. It should have all the paths set up correctly for you.

Now find your

npm/node_modules/blade

and

Meteor/packages

folders. Mine are located at

C:\Users\Mark\AppData\Roaming\npm\node_modules\blade

and

C:\Program Files (x86)\Meteor\packages

respectively.

Inside the

node_modules/blade

folder there should be another folder called “meteor”. Copy and paste this in-place to make a duplicate, then rename your copy to “blade”. We want to rename it before we move it because there is a folder in the meteor packages called “meteor” already. Now cut-and-paste your new folder into Meteor/packages.

Now go into

npm/node_modules/blade/lib

and copy all the files inside there into your newly created

Meteor/packages/blade

.

Now edit

Meteor/packages/blade/package.js

and change this line:



blade = require('../../packages/blade/node_modules/blade');

To point to the proper node module, e.g.,



blade = require('C:/Users/Mark/AppData/Roaming/npm/node_modules/blade');

That’s it.

Adding blade to your Meteor project

Just

cd

into your project folder and

meteor add blade

.

Usage is explained on the official Blade wiki.

(Sorry for the terrible formatting, those are supposed to be inline code blocks… I hate WordPress)

Posted on No Comments on Microphone Boost

Microphone Boost

This post is a bit a different than my usual posts.

I was on Skype, about to start a gaming session with my friend, when he notified me that my mic was really quiet compared to when he talked to his other friends. I checked my settings, everything was maxed out in both Skype and Windows Sound settings.

He told me about this “microphone boost” option that he used before, but I didn’t seem to have it. I thought maybe my driver was out of date?

I tried updating my driver via Windows Devices to no avail, so I tried looking for one from my hardware vendor. Problem is, I couldn’t remember who made my hardware, and I didn’t feel like popping open my computer.

I’ve used a program to give me my device information in the past, but I also couldn’t remember what it was called. I found a new one called HWiNFO which seems to do a pretty good job though. It looks like this:

From here I could see my motherboard was an Asus Sabertooth 55i. Googling “asus sabertooth 55i drivers” lead me to the driver I needed. Get the audio driver.

Unless of course, you have a dedicated sound card. Then grab the driver for that instead! I had one before, but I didn’t feel the need to plug it in for my new rig.

After installing the driver (and a mandatory reboot), I found this ugly guy on my desktop:

Doesn’t have a lot of settings (that I can see), but the volume slider wasn’t maxed out yet.

So I upped that, and then went back into my sound settings and discovered I had a new option!

Upping the Microphone Boost to just 10 dB give me a huge boost in volume output. Huzzah!

Hopefully someone else with the same problem will find this post useful.

Posted on 6 Comments on Thread-Safe Observable Priority Queue for WPF

Thread-Safe Observable Priority Queue for WPF

Building on my last post, I realized that you couldn’t push elements onto the queue from a worker thread, making it pretty much useless. However, if we dispatch the pushes back to the UI thread, it should work, right?

Here’s (what I believe to be) a thread-safe observable priority queue with notifications for use in WPF.

class PriorityQueue<TValue, TPriority> : IEnumerable<TValue>, INotifyCollectionChanged
    where TValue : INotifyPropertyChanged
    where TPriority : IComparable
{
    private SortedDictionary<TPriority, Queue<TValue>> dict;
    private int count;
    private Dispatcher dispatcher;

    public int Count { get { return count; } }
    public bool Empty { get { return Count == 0; } }

    public PriorityQueue(Dispatcher dispatcher = null)
    {
        this.dispatcher = dispatcher ?? Dispatcher.CurrentDispatcher;
        this.count = 0;
        this.dict = new SortedDictionary<TPriority, Queue<TValue>>(new ReverseComparer());
    }

    private class ReverseComparer : IComparer<TPriority>
    {
        public int Compare(TPriority x, TPriority y) { return y.CompareTo(x); }
    }

    public virtual void Push(TValue val, TPriority pri = default(TPriority))
    {
        if (dispatcher.CheckAccess())
        {
            ++count;
            if (!dict.ContainsKey(pri)) dict[pri] = new Queue<TValue>();
            dict[pri].Enqueue(val);
            OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Add, val));
        }
        else
        {
            dispatcher.Invoke(new Action<TValue, TPriority>(Push), DispatcherPriority.Send, val, pri);
        }
    }

    public virtual TValue Peek()
    {
        return dict.First().Value.Peek();
    }

    public virtual TValue Pop()
    {
        if (dispatcher.CheckAccess())
        {
            --count;
            var pair = dict.First();
            var queue = pair.Value;
            var val = queue.Dequeue();
            if (queue.Count == 0) dict.Remove(pair.Key);
            OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Remove, val));
            return val;
        }
        else
        {
            return (TValue)dispatcher.Invoke(new Func<TValue>(Pop), DispatcherPriority.Send);
        }
    }

    public event NotifyCollectionChangedEventHandler CollectionChanged;

    public virtual void OnCollectionChanged(NotifyCollectionChangedEventArgs e)
    {
        if (CollectionChanged != null)
        {
            CollectionChanged(this, e);
        }
    }

    public IEnumerator<TValue> GetEnumerator()
    {
        foreach (var queue in dict.Values)
        {
            foreach (var value in queue)
            {
                yield return value;
            }
        }
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }
}

Read this blog post to find out how this works.

Posted on No Comments on Django Send HTML Emails

Django Send HTML Emails

You can use this little function to load emails from a template and send them in both HTML and plaintext formats.

from django.core.mail import EmailMultiAlternatives
from django.template import loader, Context
from django.conf import settings

def send_multipart_email(subject, template, data_dict, recipient_list, from_email=settings.DEFAULT_FROM_EMAIL):
    if type(recipient_list) != list: recipient_list = [recipient_list]
   
    tt = loader.get_template(template+'.txt')
    c = Context(data_dict)
   
    e = EmailMultiAlternatives(subject, tt.render(c), from_email, recipient_list)
   
    try:
        ht = loader.get_template(template+'.html')
        e.attach_alternative(ht.render(c), 'text/html')
    except:
        pass
   
    e.send()
Posted on No Comments on Django, Flatpages, Markdown, and Syntax Highlighting

Django, Flatpages, Markdown, and Syntax Highlighting

For what should have been an easy task, this turned out to be extraordinarily difficult.

I assume you have Django already installed. If not, the tutorials on djangoproject.com aren’t too terrible, provided you’re not on a shared server (that’s another story!). I’m using Flatpages, but you can use whatever you want to follow along with this tutorial.

1. Install Markdown and Pygments for Python if you haven’t already. If you’re on Ubuntu, you can `sudo apt-get install python-markdown` or `easy_install Markdown` and `sudo apt-get install python-pygments`.

2. Go into your Django app and `mkdir templatetags`. This should be at the same level as `manage.py`, `settings.py` and `urls.py`. Then `cd` into that directory, and `touch __init__.py` (so that it becomes a module) and then add another file called something like `tags.py` (don’t name it “markdown.py” — that’ll bite you later). Paste this code in there:

from django import template
from django.template.defaultfilters import stringfilter

from django.conf import settings

register = template.Library()

@register.filter(name='markdown')
@stringfilter
def markdown(value, arg=''):
    """
    Filter to create HTML out of Markdown, using custom extensions.

    The diffrence between this filter and the django-internal markdown
    filter (located in ``django/contrib/markup/templatetags/markup.py``)
    is that this filter enables extensions to be load.

    Usage::

        {{ object.text|markdown }}
        {{ object.text|markdown:"save" }}
        {{ object.text|markdown:"codehilite" }}
        {{ object.text|markdown:"save,codehilite" }}

    This code is taken from
    http://www.freewisdom.org/projects/python-markdown/Django
    """
    try:
        import markdown
    except ImportError:
        if settings.DEBUG:
            raise (template.TemplateSyntaxError,
                   "Error in {% markdown %} filter: "
                   + "The markdown library isn't installed.")
        else :
            from django.utils.html import escape, linebreaks
            return linebreaks(escape(value))
    else:
        extensions=arg.split(",")
        if len(extensions) > 0 and extensions[0] == "safe" :
            extensions = extensions[1:]
            safe_mode = True
        else :
            safe_mode = False
        return markdown.markdown(value, extensions, safe_mode=safe_mode)

Yes, I’m leaving the link in there; I didn’t write that code.

3. Go into `settings.py` and add your app to the `INSTALLED_APPS`. You actually have to `import` your own app too; this baffled me for about an hour. My app is called `tech_eval` so my `INSTALLED_APPS` looks like this:

import tech_eval # you can put this line at the top of settings.py if you want

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.admin',
    'django.contrib.flatpages',
    'tech_eval',
)

4. Like I said, I’m using Flatpages, so my template looks like this (I saved it to djangoproject/app/templates/flatpages/default.html).

{% load tags %}
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
    <title>{{ flatpage.title }}</title>
    <link rel="stylesheet" type="text/css" href="/tech_eval/media/css/code_style.css" />
</head>
<body>
    <h1>{{ flatpage.title }}</h1>
    {{ flatpage.content|markdown:"codehilite"|safe}}
</body>
</html>

Notice how we filter the content with markdown, make sure the codehilite extension is used, and flag it as `safe` afterwords (otherwise Django will automatically character-encode everything).

5. Hopefully this is already working for you at this point, but you probably want to add some color to your code. You can generate the appropriate css file with this:

pygmentize -f html -S default -a .codehilite >> code_style.css

Some different styles you can use (at least on my system) are:

['manni', 'colorful', 'murphy', 'autumn', 'bw', 'pastie', 'native', 'perldoc', 'borland', 'trac', 'default', 'fruity', 'emacs', 'friendly']

(stolen from here).

Or just copy and paste this CSS file if you’re lazy:

.codehilite .c { color: #408080; font-style: italic } /* Comment */
.codehilite .err { border: 1px solid #FF0000 } /* Error */
.codehilite .k { color: #008000; font-weight: bold } /* Keyword */
.codehilite .o { color: #666666 } /* Operator */
.codehilite .cm { color: #408080; font-style: italic } /* Comment.Multiline */
.codehilite .cp { color: #BC7A00 } /* Comment.Preproc */
.codehilite .c1 { color: #408080; font-style: italic } /* Comment.Single */
.codehilite .cs { color: #408080; font-style: italic } /* Comment.Special */
.codehilite .gd { color: #A00000 } /* Generic.Deleted */
.codehilite .ge { font-style: italic } /* Generic.Emph */
.codehilite .gr { color: #FF0000 } /* Generic.Error */
.codehilite .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.codehilite .gi { color: #00A000 } /* Generic.Inserted */
.codehilite .go { color: #808080 } /* Generic.Output */
.codehilite .gp { color: #000080; font-weight: bold } /* Generic.Prompt */
.codehilite .gs { font-weight: bold } /* Generic.Strong */
.codehilite .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.codehilite .gt { color: #0040D0 } /* Generic.Traceback */
.codehilite .kc { color: #008000; font-weight: bold } /* Keyword.Constant */
.codehilite .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */
.codehilite .kp { color: #008000 } /* Keyword.Pseudo */
.codehilite .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */
.codehilite .kt { color: #B00040 } /* Keyword.Type */
.codehilite .m { color: #666666 } /* Literal.Number */
.codehilite .s { color: #BA2121 } /* Literal.String */
.codehilite .na { color: #7D9029 } /* Name.Attribute */
.codehilite .nb { color: #008000 } /* Name.Builtin */
.codehilite .nc { color: #0000FF; font-weight: bold } /* Name.Class */
.codehilite .no { color: #880000 } /* Name.Constant */
.codehilite .nd { color: #AA22FF } /* Name.Decorator */
.codehilite .ni { color: #999999; font-weight: bold } /* Name.Entity */
.codehilite .ne { color: #D2413A; font-weight: bold } /* Name.Exception */
.codehilite .nf { color: #0000FF } /* Name.Function */
.codehilite .nl { color: #A0A000 } /* Name.Label */
.codehilite .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
.codehilite .nt { color: #008000; font-weight: bold } /* Name.Tag */
.codehilite .nv { color: #19177C } /* Name.Variable */
.codehilite .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */
.codehilite .w { color: #bbbbbb } /* Text.Whitespace */
.codehilite .mf { color: #666666 } /* Literal.Number.Float */
.codehilite .mh { color: #666666 } /* Literal.Number.Hex */
.codehilite .mi { color: #666666 } /* Literal.Number.Integer */
.codehilite .mo { color: #666666 } /* Literal.Number.Oct */
.codehilite .sb { color: #BA2121 } /* Literal.String.Backtick */
.codehilite .sc { color: #BA2121 } /* Literal.String.Char */
.codehilite .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */
.codehilite .s2 { color: #BA2121 } /* Literal.String.Double */
.codehilite .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */
.codehilite .sh { color: #BA2121 } /* Literal.String.Heredoc */
.codehilite .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */
.codehilite .sx { color: #008000 } /* Literal.String.Other */
.codehilite .sr { color: #BB6688 } /* Literal.String.Regex */
.codehilite .s1 { color: #BA2121 } /* Literal.String.Single */
.codehilite .ss { color: #19177C } /* Literal.String.Symbol */
.codehilite .bp { color: #008000 } /* Name.Builtin.Pseudo */
.codehilite .vc { color: #19177C } /* Name.Variable.Class */
.codehilite .vg { color: #19177C } /* Name.Variable.Global */
.codehilite .vi { color: #19177C } /* Name.Variable.Instance */
.codehilite .il { color: #666666 } /* Literal.Number.Integer.Long */
.codehilitetable { border: 1px solid #666; width: 100%; background-color: #666; }
.codehilitetable .linenos { background-color: #666; padding: 0 5px; width: 20px; }
.codehilitetable .code { padding: 0 15px; background-color: #fff; }

And that should pretty much be it. Let me know if I missed anything or this was too difficult to follow along.

Posted on 2 Comments on Insert into MySQL datetime column from PHP

Insert into MySQL datetime column from PHP

You can use this simple function to convert a unix timestamp (like the one obtained from time()) to MySQL datetime format:

function mysql_datetime($timestamp = null) {
    if(!isset($timestamp)) $timestamp = time();
    return date('Y-m-d H:i:s', $timestamp);
}

I’m using this instead of MySQL’s NOW() function because my MySQL server isn’t localized to my timezone, but my PHP is by usage of date_default_timezone_set.

Edit: And you can use this function to convert back the other way (you can do this directly in the SQL too, but just in case)

function datetime_to_unix($datetime) {
    list($date, $time) = explode(' ', $datetime);
    list($year, $month, $day) = explode('-', $date);
    list($hour, $minute, $second) = explode(':', $time);
    return mktime($hour, $minute, $second, $month, $day, $year);
}
Posted on No Comments on Simple Rounded Corners with CSS

Simple Rounded Corners with CSS

I know there are umpteen billion tutorials on rounded corners out there, but here’s an easy way that I like to do it. Requires only one image, and allows you to nudge your text as close to the corners are you like.

If you want a 3px radius corner, have a black background, and white foreground, you need to create a 6×6 (i.e. radius*2) image of a white circle on a black background. This method won’t support transparency unfortunately.

The HTML:

<div id="content">
    <div id="tl"></div><div id="tr"></div><div id="bl"></div><div id="br"></div>
    Your content here.
</div>

The CSS:

#content {
    position:relative;
    background-color: #fff;
    width:600px;
    height:300px;
}

#tl,#tr,#bl,#br{
    background: #000 url(images/circle.gif);
    width:3px;
    height:3px;
    position:absolute;
}

#tl{top:0;left:0}
#tr{top:0;right:0;background-position:3px 0}
#bl{bottom:0;left:0;background-position:0 3px}
#br{bottom:0;right:0;background-position:3px 3px}

Adjust all the “3px”s to match your radius, and modify the background attribute to meet your needs, and style content however you want. You probably want to put some padding on there. The important thing is that #content has position:relative so that the corners are aligned to the content box rather than the page.

I used this technique on my personal site, mnbayazit.com.