Archiving the Web

July 16, 2012 at 4:51 pm | Posted in Internet, Media, Online services, Software | Leave a comment

If you’re a researcher or writer, you’ve probably run into the issue of a web reference vanishing. Some sites treat articles as temporary. It’s one of the reasons reference standards for web citations include the date found. But digital information is so much superior to paper. The key is just in properly capturing that passing data.

Tools like Evernote for capturing snippets of the web have been around for years. It captures content with links, gathering the bits in a common scroll.

I found I liked to capture the context as well, so I tended to save the whole page or article. Web browsers all allow you to simply File/Save Page As or File/Save As. It will typically use the Page Title tag as the filename. However, what you end up with is an HTML file and a name-matched folder of parts – the images, css, and so forth called by the page. Because Windows sorts folders and files separately, it’s a little too easy for the parts to get separated. If you move the file and not the folder, rename the folder, rename the file (thus visually disassociating it from the folder), or whatever, the link can be easily broken. Long titles can create problems for backup and burning software too.

Web Archive formats to the rescue. The first I discovered years ago was the .MHT format. This is a Microsoft-developed format that archives the page and its parts in an open-standard (surprise!) zip file. It opens normally in IE on a double click. To use it, just choose Web Archive from the Save As options in IE and voila, your page is packed in a single file.

However, as I used Firefox for most browsing, this was inconvenient. I happily discovered someone had built a plug-in for saving pages to MHT in Firefox. I used it for years, even editing it after it ceased being updated to keep it working.

Eventually, I came to print pages to PDF. With the right Add-on, this is as simple as File, Print to PDF. (PDF’s are “printed” as they use the post-script print engine to convert to a PDF file)

There are a number of related tools like Print Edit, which allows you to delete unnecessary parts before saving or printing.

(as an aside, for PDFs I’ve become a big fan of the free Foxit Reader. It’s way faster than Acrobat Reader and doesn’t have the security issues.)

However, PDFs are designed for Office documents, books and such. Some web pages don’t sit very well in a paper layout and other features can be broken, like embedded media.

Recently, I discovered the new Add-on Mozilla Archive Format. It allows you to save in both the MHT format mentioned above (and have them open in Firefox) and in the Mozilla Archive Format (MAFF). For MAFF, the saved page is “faithful” to the original. The format is compressed so uses less disc space, can include embedded video & audio, is an open standard and also uses universal ZIP. It even allows you to save multiple tabs in a single file. For either MHT or MAFF, it displays where you saved it from and when; perfect for references. And you can convert to other formats. As a Zip file, if you want to extract just one part like an image, you can use a Zip tool like the free 7-Zip.

Of course, no discussion of “web archives” would be complete without mentioning the Internet Archive. It’s a massive site that is archiving portions of the web. The WaybackMachine, for example, will show you old versions of many web sites. They’re also archiving many old texts, live music, and so much more. What they have is a little random but some is amazing.

May your own archive become quite the treasure also.

PS – don’t forget to have a Backup of those nice archives.

Electrifying News

September 12, 2008 at 5:37 pm | Posted in Economoney, Technology, Transportation | Leave a comment

There are lots of articles about new automotive technology floating around – from cars that run on water, to odd hybrids. Great ideas but nothing that will actually change anything enough to make a difference. The cover of the latest Wired magazine seemed at first glance to be yet another such story.

Driven tells the story of Shai Agassi’s journey from SAP into a company called Better Place. The goal is quite simply to completely change the transportation infrastructure of the world, from oil to electricity. He clearly saw that making a few electric cars was not going to cut it. There needed to be a complete infrastructure so people could charge up easily, swap batteries for a quick recharge, and save money doing it.

His vision was quite comprehensive and practical. But was it another pie in the sky? To create an entire infrastructure would need buy-in from large scale players, like the auto industry and government. Most remarkably, he’s got it. Israel and Denmark are on board, along with Renault. They’re building a beta system, complete with “AutOS”: “The system serves as energy monitor, GPS unit, help center, and personal assistant, packed into an onboard PC that will also hold cellular and Wi-Fi chips.”

And the cars are no flakes or downgrades.

Notable however is that the US is resistant. As Al Gore noted in An Inconvenient Truth, while the US is proud of it’s cars, they have actually fallen behind many other countries. Stories like the film Who Killed the Electric Car document how they have even back-pedaled on successful initiatives.

The article is well worth the read, if even in seeing how a vision can get built. This will be a story to follow.


Related articles can be found here.

Gore Redux

April 14, 2008 at 9:28 am | Posted in Economoney, Health, Movies, Nature, Science | Leave a comment

In Feb 2006, Al Gore gave a short follow-up to his famous film “An Inconvenient Truth” at the remarkable TED talks. (on Ideas) He briefly outlines what you can do. (17′)
In his own words, he “used to be the next President of the United States of America” but has since changed professions.

2 years later, in Feb this year, in this obviously rushed and brand new presentation at TED, he focuses on the need for more than new lightbulbs. Some of the climate changes are accelerating faster than predicted and we need to take broader strokes faster. He observes how they made the climate an issue in the recent Australian elections (helped by a serious drought) and ratifying Kyoto* became one of the first actions of the new government. The only remaining Kyoto holdout is the US. Guess what they plan for this years elections. (29′)
This is the one to show people who have not see “An Inconvenient Truth”. A little rushed and he shows fewer consequences, but comparing Earth to Venus is interesting.

If you think this is just his opinion, he shared the 2007 Nobel Peace Prize with the Intergovernmental Panel on Climate Change for his ‘new profession’. You have to keep in mind that there is a lot of noise out there trying to give the impression there is widespread scientific debate on the issue. Certainly, one can debate some “facts” as they are projections, like trying to predict the weather. But whats happening to our climate is undebatable and off the chart on any known historical cycles, going back many thousands of years. We need to do what we can to mitigate our contribution to the problem. At the very least, this will make our towns a cleaner, healthier place. And it does NOT have to cost a lot. Indeed, the cost of not acting is much greater. The longer we wait the harder it will be to improve.

The speed of change is shocking scientists. Evidently, the poles are warming much faster than further south, yet they can have a profound impact globally. Researchers barely needing an icebreaker in November, unlike the famous journey of the St Roch, still within some peoples lifetime.  Another example, the new skirmishes over the territory in Canada’s northern waters. Other countries are now trying to lay claim to what was once uninhabitable. Add in the consequences for wildlife.   (a slide show)

The importance of the Arctic ice cannot be underestimated. The seasonal patterns of the ice and water flow from the area drive the global ocean currents which in turn drive the weather. Historically, the last time the Arctic ice opened up this much, Europe had a mini ice age. Keep in mind that London is further north than Calgary in Canada. It is the northern flowing warm water that moderates their climate, driven by the southern flowing cool water off western Greenland.

The burgeoning green industry sector tells us going green is not going to destroy jobs. Its going to create them. But it does require change. Change for the better. Its only rigid thinking that considers that a bad thing.


*The Kyoto Accord is a very basic level of reduction in emissions the world actually was able to agree on, simply correcting to levels they previously were at in 1990, but a few countries were unwilling to ratify the agreement. The US is the last holdout, even after their own bad experiences with climate change, like Katrina. As a result of this, other countries are waiting or backsliding, like Canada.

11th Hour

September 15, 2007 at 1:09 am | Posted in Movies, Science | Leave a comment

Saw the film “11th Hour” this evening.  The film “an inconvenient truth” is mainly one guy talking, with some powerful graphics. Unexpectedly effective. 11th Hour is a whole series of talking heads – from a former CIA director to environmentalists, interspersed with short clips of the beauty of the earth and the devastation we have wrought.

Koyansquatsi, the early ’80’s film “life out of balance” (with an amazing musical score)  highlighted how we were drifting to an extreme. And that was 25 years ago. We didn’t get the message then and the issues are accelerating.

If you are a boomer, the worlds population has more than doubled in your lifetime. At the same time, consumption by “first world” countries has increased dramatically. We have become more familiar with brand logos than local plant species.

It all comes down to a few false assumptions. That we are separate from nature.  And that if more is good, even more is better. For many people, our lives have become a never ending rat race of working, just to buy stuff. Our entire culture has become built around these basic and very false ideas. We are not separate from nature and as the film observes, if we don’t come to recognize our role in the larger global context, we willl perish as have the majority of specifies before us. Just look at the epidemic of obesity for how we treat ourselves.

Though not mentioned in the film, the sudden decline of bee populations is another fine example. The majority of our crops depend on the bees. If they die out, these food crops will fail within 4 years.  Einstein made that observation many years ago.

The film highlights the role of unfettered capitalism and its influence over government policy. Industrialization was originally developed to improve our quality of life but has passed a tipping point, a place where it is reducing quality of life. Corporations pursuing more profits at any cost become a monster, destroying the people they were once intended to serve.

Buckminster Fuller many years ago said that the key to a high quality of living was low cost energy.  Because energy was the ability to do work. He proposed (amonst many other things), a global electrical grid to share power where peak demands were distributed and sources shared.

As the film observes, oil became our ticket to cheap energy. Only trouble is, it pollutes, and is a non-renewable resource. In coal and oil, we are borrowing from the past and using the resource at a far greater rate than the supply can sustain.

Some powerful points in there about how we view everything as either a person (corporations included) or as property  – all of nature.  But nature is not our property. It is not segmented and cannot be viewed outside the bigger picture.

The film only breezes over some of the projections and observations of existing consequences. But it tries hard to stay with facts and not conjecture. Its a powerful indictments of our recent way of life.

The film also touches on solutions. That the technology for cheap and clean energy is here. That the majority of people want it.  Only that the largest corporations and their government supporters are not interested in change. The film “Who Killed the Electric Car” documents how this plays out – the electric cars  on the market 10 years ago are all off the market now.

The film also notes how Republicans and Democrats in the US came together to introduce a series of environmental acts many years ago, most of which has since been disabled.

The wisdom is there. We’ve taken a kick at the can a few times, but as the film observes, we’re running out of time. We don’t know for what. But if we continue to treat our environment (and ourselves) with abuse and disdain, nature will do what it has so many times before and eradicate the infection.

We can as individuals make a difference – at least to slow the acceleration and give us some choices. 

Blog at
Entries and comments feeds.