I had constantly worked on improving the site load time since last few weeks and you might have noticed some of the changes as well. If you are a regular reader, its hard to not notice them. :) As per my experience, I have made a list of things that you can do in order to improve your WordPress site's load time.
Server Response Time
If your host is unreliable and responds slowly, you won't be able to improve site speed. So, first, get a good paid host. I can't speak for all, but most of the free hosts are unreliable and have slow response time, and yes, there are even paid hosts that are slow. So, choose your host wisely.
Use cache plugins such as WP Super Cache or W3 Total Cache. Although, cache plugins do not help improve site load time for new visitors, they are certainly helpful for improving load time for regular readers. Cache plugins help store recently visited page to the computer's memory, so that the contents of the page do not need to be retrieved from the server again. They are simply retrieved from the temporary storage on the visitor's computer. This helps improve site load time, and saves bandwidth as well.
This change has obviously been noticed by the regular readers here. I have changed the site's theme to give it a clean and minimalistic look. If you have a heavy theme, it will have a negative impact on your site speed. So, while choosing the themes for your site, I strongly urge you to go for a minimal theme.
Lazy Load Images
Lazy Loading is when you load the contents lazily, meaning, you don't load everything at once, but when it's the right time to load them. If you aren't currently lazy loading the images, then your page will make lots of HTTP requests at once. Do this : open up one of your posts which has got a few images.Notice how all of those images will be retrieved at the same time.
But, if you lazy load the images, they will be loaded only when the reader scrolls down to the area of your post where the images are placed. This will improve your page load time since the heavy stuff on your site (i.e. the images) doesn't get loaded when you open the page. I'm using Bj Lazy Load plugin at the moment, but you can find several alternatives on the WordPress plugin repository.
Smush.it is a service provided by Yahoo! which reduces image file's size. It is a completely lossless way of optimizing images, which means that the service does not degrade original image's quality. Luckily, there's a WordPress plugin that uses Smush.it API to automatically optimize images as soon as they are uploaded. You can get it from here. When the image's size is decreased, the site load time also improves.
Lazy Load Comments
I had recently made changes on the site to lazy load comments. Just like images, comments can also be lazy loaded. If you load all the comments at once, your site has to make more requests in order to retrieve the comments and the gravatar images associated with them. I don't know how to lazy load comments using the native WordPress comment feature, but what I did was replace it with Disqus Comment System and then lazy-load the comments.
There's two ways you can do it : Don't load the comments at first. Load comments only when a button is clicked. Or, load the comments when visitor scrolls halfway to the page. You can find the guide to the latter one online, just google for it.
Lazy Load Social Sharing Buttons
Social Sharing Buttons are the main culprits for increased site load time. Here's a way for you to check this : Analyze your page size and requests made. Then, disable the social sharing button/widgets that you have currently installed on your site, and then re-check your page size and requests. (Check size/requests of a post). You should notice the difference.
Earlier I had installed Digg Digg plugin which showed a floating social sharing panel on the post. I was stunned to know that Digg Digg was responsible for increasing the page's size by 400-500 Kb. That's a lot of increase.
If you feel the same, what you can do is, use WPSocialite plugin. It places the icons of the social sites on your post, and when someone hovers on the icon/content, it then loads the sharing buttons. That's lazy load for you.
Remove Unused/Unneccessary Plugins & Widgets
Need I say anything about this? Even inactive plugins can have files that can effect your site load time. Besides, even though the plugins are inactive, they can be vulnerable to different kinds of attacks, so its always good to remove them completely. Try to use less number of widgets. If you think removing a widget or two won't hamper your site then do so.
That's all I did to improve the site's speed. There are other things you can do like lazy loading the widgets, or using less number of images on the post etc. Also, note that all the things don't go well together. For example, while I was using W3 Total Cache plugin, my site speed had degraded. I switched to WP Super Cache and the problem was solved. But, right now, I am having better results by not using any of the cache plugins.
So, do your homework and find out what goes good with your site by analyzing the size and load time using tools such as GtMetrix, Pingdom etc. Good Luck!