Slow is annoying.
People like things fast. Fast cars, fast good, and fast downloads. Who wouldn't? There's nothing more annoying than sitting on your phone and waiting for something to load. And if something takes too long to load, we move on. That's why performance of a website is so important.
We know from surveys that people expect pages to load in two seconds or less. That's our definition of fast because it's what our users expect. We also know that nearly half of people say they'll abandon a page that takes longer than four seconds to load. In actually testing we know they're a little more patient than that, but if a page takes longer than five seconds, we can expect about half of people to abandon the page.
Your bounce rate isn't going to reflect people who leave due to excessive load times. If the user gave up mid-download, your Google Analytics snippet probably never even had time to load. (Your javascripts are in your footer, right?) Using tools like Pingdom's Full Page Test, Webpagetest, or YSlow.
For that reason we need to consider performance as part of design when we create anything for the web. The reasoning behind it is simple: the faster our page loads, the more people that view our site, and the happier they'll be. More happy users means more conversions.
There are three approaches one can take to reducing the page load on the site. The most direct is to reduce the file size which we can do by stripping out content, or by compressing existing content via minification or gzip. First ask, "Do we really need this?" and then ask "How do we make what's left smaller?" Editing and then compressing content is fairly straight forward because we're just making it smaller.
We can also reduce load time by reducing overhead. In addition to considering the page size we also need to consider the total number of requests made by a page. Each request has its own overhead in that the browser has to handshake with the server, and the server can only handle so many concurrent requests. That overhead gets worse when the request is to a separate server because now it also includes an additional DNS lookup. Solving this is easier than it sounds. We can combine style sheets in to a single file, combine javascripts in to a single file, and even combine images. By creating "sprites," we can load a group of related images in to a single filmstrip and then "crop" it to just the relevant image using CSS. For site's that use a lot of SVG icons (you're using SVGs, right?) we can load them all at once in an icon font. We generate ours using IcoMoon.
Now that we've gotten our page down as small as it will go, and we've reduced our total number of requests, we can take our third and final approach which is to make our page render faster. By altering the order in which things load we'll give our site the appearance that it's loading faster. Browsers render things in the order they appear on the page. We can exploit that by loading our stylesheet first in the site's head which will render the site faster. If we move our JavaScript to to bottom of the page after the body tag, we'll add our JS functionality only after the site is rendered. Doing this doesn't change the size of the page but it will have the effect of making the page render faster. In some browsers we can even take this a step further and use the "async" and "defer" attributes on javascript. Async will specify that a file can be loaded asynchronously, and defer will specify that a file should be loaded only after the rest of the page is loaded. The final tool in our belt is to use good old progressive JPEGs instead of optimized JPEGs. Sometimes they'll be a little larger, but they render faster so they'll appear to load faster.
These are all things an experienced developer should be doing to begin with. Websites that load fast aren't just best practice, they're fun. Tweaking a page to reduce its load time is like playing golf. You want get your score as low as possible through careful optimization of techniques.